Datasets

You can download individual data files or bulk download entire universes of assets. For more information about each of these data formats, see Downloading Data.

When you download data from the Dataset Market, it's stored in the data directory in your organization workspace. This is the same directory that LEAN reads data from when you run an algorithm or spin up a research notebook. To change the directory from which LEAN reads the data, open the configuration file and adjust the value under the data-folder key. The data folder should be fast and spacious. Follow these steps to check the approximate size of the datasets in the Dataset Market:

1. Open the Download in Bulk page.
2. Click a dataset.
3. Scroll down to the Size and Format section.

LEAN strives to use an open, human-readable format, so all data is stored in flat files (formatted as CSV or JSON). The data is compressed on disk using zip. For more information about general data formats, see the LEAN Data Formats README in the LEAN GitHub repository. The following README files explain the data formats of specific asset classes:

• Equity
• Equity Options
• Crypto
• Forex
• Futures
• CFD

You need live data providers to inject data into your algorithm so that you can make real-time trading decisions and so that the values of the securities in your portfolio are updated. The data providers you have available to you depend on where you deploy the algorithm. When you request historical data in local algorithms, the historical data comes from your dataset vendor. Familiarize yourself with the quotas and limits of your data provider to avoid errors.

Local Deployments

When you deploy local algorithms, you can use any of the following data providers:

• A brokerage data provider

These are streams of live security prices that come directly from your brokerage.

• IQFeed

To view the asset classes that our IQFeed integration supports, see Supported Assets.

• Polygon

To view the asset classes that our Polygon integration supports, see Supported Assets.

Cloud Deployments

When you deploy algorithms to QuantConnect Cloud, you can use any of the data providers we support in the cloud. Your live algorithms run on our co-located servers that have 10 GB transfer speeds and low latency.