Importing Data
Key Concepts
Introduction
Custom data is your own external data that's not from the Dataset Market. You can use custom data to inform trading decisions and to simulate trades on unsupported securities. To get custom data into your algorithms, you download the entire file at once or read it line-by-line with a custom data reader. If you use a custom data reader, LEAN sends the data to the OnData
method in your algorithm.
Stream Custom Data
To receive your custom data in the OnData
method, create a custom type and then create a data subscription. The custom data type tells LEAN where to get your data and how to read it.
All custom data types must extend the BaseData
PythonData
class and override the GetSource
and Reader
methods
public class MyCustomDataType : BaseData { public override DateTime EndTime { get; set; } public decimal Property1 { get; set; } = 0; public override SubscriptionDataSource GetSource( SubscriptionDataConfig config, DateTime date, bool isLive) { return new SubscriptionDataSource("<sourceURL>", SubscriptionTransportMedium.RemoteFile); } public override BaseData Reader( SubscriptionDataConfig config, string line, DateTime date, bool isLive) { if (string.IsNullOrWhiteSpace(line.Trim()) || char.IsDigit(line[0])) { return null; } var data = line.Split(','); return new MyCustomDataType() { Time = DateTime.ParseExact(data[0], "yyyyMMdd", CultureInfo.InvariantCulture), EndTime = Time.AddDays(1), Symbol = config.Symbol, Value = data[1].IfNotNullOrEmpty( s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture)), Property1 = data[2].IfNotNullOrEmpty( s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture)) }; } }
class MyCustomDataType(PythonData): def GetSource(self, config: SubscriptionDataConfig, date: datetime, isLive: bool) -> SubscriptionDataSource: return SubscriptionDataSource("<sourceURL>", SubscriptionTransportMedium.RemoteFile) def Reader(self, config: SubscriptionDataConfig, line: str, date: datetime, isLive: bool) -> BaseData: if not (line.strip() and line[0].isdigit()): return None data = line.split(',') custom = MyCustomDataType() custom.Time = datetime.strptime(data[0], '%Y%m%d') custom.EndTime = custom.Time + timedelta(1) custom.Value = float(data[1]) custom["Property1"] = float(data[2]) return custom
For more information about custom data types, see Streaming Data.
Download Bulk Data
The Download
method downloads the content served from a local file or URL and then returns it as a string.
var content = Download("<filePathOrURL>");
content = self.Download("<filePathOrURL>")
For more information about bulk downloads, see Bulk Downloads.
Remote File Providers
The most common remote file providers to use are Dropbox, GitHub, and Google Sheets.
Dropbox
If you store your custom data in Dropbox, you need to create a link to the file and add ?dl=1
to the end of the file URL. To create file links, see How to share files or folders in the Dropbox documentation. The ?dl=1
parameter lets you download the direct file link, not the HTML page of the file.
GitHub
If you store your custom data in GitHub, you can use public or private repositories. When you download the data, use the raw file (for example, https://raw.githubusercontent.com/<organization>/<repo>/<path>). For instructions on accessing the raw file, see Viewing or copying the raw file content in the GitHub documentation.
Google Sheets
If you store your custom data in Google Sheets, you need to create a link to the file. To create file links, see Make Google Docs, Sheets, Slides & Forms public in the Google Docs documentation. Choose the CSV format to find /pub?gid={SHEET_ID}&single=true&output=csv
parameter on the end of the link. Google Sheets don't support exporting data in JSON format.
File Quotas
The following table shows the number of files you can download during a single backtest or Research Environment session in QuantConnect Cloud:
Organization Tier | File Quota |
---|---|
Free | 25 |
Quant Researcher | 100 |
Team | 250 |
Trading Firm | 1,000 |
Free | Unlimited |
Each file can be up to 200 MB in size and have a file name up to 200 characters long.
If you need to import more files than your quota allows, save your custom data files in the Object Store and load them from there.