--- title: "Rendering Data with Python" description: "This page explains how to create a script to download and process your dataset with Python for QuantConnect distribution." canonical: https://www.quantconnect.com/docs/v2/lean-engine/contributions/datasets/rendering-data/rendering-data-with-python markdown: https://www.quantconnect.com/docs/v2/lean-engine/contributions/datasets/rendering-data/rendering-data-with-python.md language: python source: https://github.com/QuantConnect/Documentation/tree/master/06%20LEAN%20Engine/02%20Contributions/01%20Datasets/04%20Rendering%20Data/00%20Rendering%20Data%20with%20Python modified: 2026-10-29 index: https://www.quantconnect.com/llms.txt --- # Rendering Data with Python [Documentation](https://www.quantconnect.com/docs/v2.md) > [LEAN Engine](https://www.quantconnect.com/docs/v2/lean-engine.md) > Contributions > Datasets > Rendering Data > Rendering Data with Python > Code examples are shown in Python. Append `?lang=csharp` to this URL for the C# version of this page. ## Introduction This page explains how to create a script to download and process your dataset with Python for QuantConnect distribution. ## Using Processing Framework During this part of the contribution process, you need to edit the `Lean.DataSource. / DataProcessing / process.py` file so it transforms and moves your raw data into the format and location the [GetSource methods](https://www.quantconnect.com/docs/v2/lean-engine/contributions/datasets/defining-data-models.md) expect. The script should save all the data history to the `output` directory in your machine's root directory (for example, `C: / output`) and it should save a sample of the data history to the `Lean.DataSource. / output` directory. Follow these steps to set up the downloading and processing script for your dataset: 1. Change the structure of the `Lean.DataSource. / output` directory to match the path structure you defined in the `get_source` methods (for example, `output / alternative / xyzairline / ticketsales`). 2. In the `Lean.DataSource. / DataProcessing / process.py` file, add some code to time how long it takes to process the entire dataset and how long it takes to update the dataset with one day's worth of data. > You need this information for when you provide the [dataset documentation](https://www.quantconnect.com/docs/v2/lean-engine/contributions/datasets/data-documentation.md). We need to know how long it takes to process your dataset so we can schedule its processing job. 3. In the processing file, load the raw data from your source. You can fetch data from any of the following sources: | Source | Considerations | | --- | --- | | Local Files | It can help to first copy the data into location. | | Remote API | Stay within the rate limits. You can use the rate gate class. | You should load and process the data period by period. Use the date range provided to the script to process the specific dates provided. 4. If your dataset is for universe selection data and it's at a higher frequency than hour resolution, resample your data to hourly or daily resolution. 5. If any of the following statements are true, skip the rest of the steps in this tutorial: - Your dataset is not linked to Equities. - Your dataset is related to Equities and already includes the point-in-time tickers. If your dataset is related to Equities and your dataset doesn't account for ticker changes, the rest of the steps help you to adjust the tickers over the historical data so they are point-in-time. - If you don't have the [US Equity Security Master dataset](https://www.quantconnect.com/datasets/quantconnect-us-equity-security-master/cli), [contact us](https://www.quantconnect.com/contact). - In the `Lean.DataSource. / DataProcessing / Program.cs` file, remove the statements of the `Main` method - In a terminal, compile the data processing project. ``` $ dotnet build .\DataProcessing\DataProcessing.csproj ``` This step generates a file that the `CLRImports` library uses. - In the `Lean.DataSource. / DataProcessing / process.py` file, import the `CLRImports` library. ```python from CLRImports import * ``` - Create and initialize a map file provider. ```python map_file_provider = LocalZipMapFileProvider() map_file_provider.initialize(DefaultDataProvider()) ``` - Create a security identifier. ```python sid = SecurityIdentifier.generate_equity(point_in_time_ticker, Market.USA, True, map_file_provider, csv_date) ``` - Copy the `process.py` script to the `DataProcessing / bin / debug / net9.0` directory. You need to place the script under the `bin` directory so that LEAN's packages dlls are correctly loaded for the `CLRImports`. ``` $ cp process.py DataProcessing/bin/Debug/net9.0 ``` - Run the `DataProcessing / bin / debug / net9.0 / process.py` script to populate the `Lean.DataSource. / output` directory and the `output` directory in your machine's root directory. ``` $ cd DataProcessing/bin/debug/net9.0/ $ python process.py ``` > Note: The pull request you make at the end must contain sample data so we can review it and run the demonstration algorithms. ## Python Processor Examples The following examples are rendering datasets with Python processing: - [Lean.DataSource.BitcoinMetadata](https://github.com/QuantConnect/Lean.DataSource.BitcoinMetadata/blob/master/process.py) - [Lean.DataSource.BrainSentiment](https://github.com/QuantConnect/Lean.DataSource.BrainSentiment/blob/master/DataProcessing/process.py) - [Lean.DataSource.CryptoSlamNFTSale](https://github.com/QuantConnect/Lean.DataSource.CryptoSlamNFTSale/blob/master/process.py) - [Lean.DataSource.QuiverQuantTwitterFollowers](https://github.com/QuantConnect/Lean.DataSource.QuiverQuantTwitterFollowers/blob/master/DataProcessing/process.py) - [Lean.DataSource.Regalytics](https://github.com/QuantConnect/Lean.DataSource.Regalytics/blob/master/process.py)