Rendering Data

During this part of the contribution process, you need to edit the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / process.sample.py file so it transforms and moves your raw data into the format and location the GetSource methods expect. The script should save all the data history to the output directory in your machine's root directory (for example, C: / output) and it should save a sample of the data history to the Lean.DataSource.<vendorNameDatasetName> / output directory.

Follow these steps to set up the downloading and processing script for your dataset:

1. Change the structure of the Lean.DataSource.<vendorNameDatasetName> / output directory to match the path structure you defined in the GetSource methods (for example, output / alternative / xyzairline / ticketsales).
2. In the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / process.sample.py file, add some code to time how long it takes to process the entire dataset and how long it takes to update the dataset with one day's worth of data.
You need this information for when you provide the dataset documentation. We need to know how long it takes to process your dataset so we can schedule its processing job.
3. In the processing file, load the raw data from your source.

You can fetch data from any of the following sources:

Source	Considerations
Local Files	It can help to first copy the data into location.
Remote API	Stay within the rate limits. You can use the rate gate class.

You should load and process the data period by period. Use the date range provided to the script to process the specific dates provided.

4. If your dataset is for universe selection data and it's at a higher frequency than hour resolution, resample your data to hourly or daily resolution.
   
5. If any of the following statements are true, skip the rest of the steps in this tutorial:
• Your dataset is not linked to Equities.
• Your dataset is related to Equities and already includes the point-in-time tickers.

If your dataset is related to Equities and your dataset doesn't account for ticker changes, the rest of the steps help you to adjust the tickers over the historical data so they are point-in-time.

6. If you don't have the US Equity Security Master dataset, contact us.
7. In the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / Program.cs file, remove the statements of the Main method
8. In a terminal, compile the data processing project.

$ dotnet build .\DataProcessing\DataProcessing.csproj

This step generates a file that the CLRImports library uses.

9. In the Lean.DataSource.<vendorNameDatasetName> / DataProcessing / process.sample.py file, import the CLRImports library.

from CLRImports import *

10. Create and initialize a map file provider.

map_file_provider = LocalZipMapFileProvider()
map_file_provider.Initialize(DefaultDataProvider())

11. Create a security identifier.

sid = SecurityIdentifier.generate_equity(point_in_time_ticker,
    Market.USA, True, map_file_provider, csv_date)

After you finish editing the process.sample.py script, run it to populate the Lean.DataSource.<vendorNameDatasetName> / output directory and the output directory in your machine's root directory.

$ python process.sample.py

The following examples are rendering datasets with Python processing:

• Lean.DataSource.BitcoinMetadata
• Lean.DataSource.BrainSentiment
• Lean.DataSource.CryptoSlamNFTSale
• Lean.DataSource.QuiverQuantTwitterFollowers
• Lean.DataSource.Regalytics