Rendering Data

During this part of the contribution process, you need to edit the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / Program.cs file so it transforms and moves your raw data into the format and location the GetSource methods expect. The program should save all the data history to the output directory in your machine's root directory (for example, C: / output) and it should save a sample of the data history to the Lean.DataSource.<vendorNameDatasetName> / output directory.

Follow these steps to set up the downloading and processing script for your dataset:

1. Change the structure of the Lean.DataSource.<vendorNameDatasetName> / output directory to match the path structure you defined in the GetSource methods (for example, output / alternative / xyzairline / ticketsales).
2. In the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / Program.cs file, add some code to time how long it takes to process the entire dataset and how long it takes to update the dataset with one day's worth of data.
You need this information for when you provide the dataset documentation. We need to know how long it takes to process your dataset so we can schedule its processing job.
3. In the processing file, load the raw data from your source.

You can fetch data from any of the following sources:

Source	Considerations
Local Files	It can help to first copy the data into location.
Remote API	Stay within the rate limits. You can use the rate gate class.

You should load and process the data period by period. Use the date range provided to the script to process the specific dates provided.

4. If your dataset is for universe selection data and it's at a higher frequency than hour resolution, resample your data to hourly or daily resolution.
   
5. If any of the following statements are true, skip the rest of the steps in this tutorial:
• Your dataset is not related to Equities.
• Your dataset is related to Equities and already includes the point-in-time tickers.

If your dataset is related to Equities and your dataset doesn't account for ticker changes, the rest of the steps help you to adjust the tickers over the historical data so they are point-in-time.

6. If you don't have the US Equity Security Master dataset, contact us.
7. In the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / Program.cs file, create and initialize a map file provider.

var mapFileProvider = new LocalZipMapFileProvider();
var mapFileProvider.Initialize(new DefaultDataProvider());

8. Create a security identifier.

var sid = SecurityIdentifier.GenerateEquity(pointInIimeTicker,
    Market.USA, true, mapFileProvider, csvDate)

9. In a terminal, compile the data processing project to generate the process.exe executable file.

   $ dotnet build .\DataProcessing\DataProcessing.csproj

After you finish compiling the Program.cs file, run the process.exe file to populate the Lean.DataSource.<vendorNameDatasetName> / output directory and the output directory in your machine's root directory.

The following examples are rendering datasets with C# processing:

• Lean.DataSource.BinanceFundingRate
• Lean.DataSource.CoinGecko
• Lean.DataSource.CryptoCoarseFundamentalUniverse
• Lean.DataSource.QuiverInsiderTrading
• Lean.DataSource.VIXCentral