Data Format
Key Concepts
Folder Structure
LEAN organizes data files into folders by security type, market, and resolution. The following table shows the security type folder of each asset class:
| Asset Class | Folder |
|---|---|
| US Equity | equity |
| Equity Options | option |
| Index | index |
| Index Options | indexoption |
| Futures | future |
| Future Options | futureoption |
| Forex | forex |
| CFD | cfd |
| Crypto | crypto |
| Crypto Futures | cryptofuture |
The market folder separates assets that have the same ticker but trade on different exchanges or brokerages. For example, BTCUSDT trades on several exchanges at slightly different prices, so each exchange has its own folder under crypto.
In general, tick, second, and minute data have one zip file for each ticker and day, in a ticker folder. Hour and daily data have one zip file for many days, in the resolution folder. The following list shows the general path patterns:
- Tick, second, and minute: <securityType> / <market> / <resolution> / <ticker> / <date>_<tickType>.zip
- Hour and daily: <securityType> / <market> / <resolution> / <ticker>.zip
Options, Futures, and Crypto add more fields to these paths, and some zip files hold one CSV file for each contract. Each asset class page gives the exact paths. The following table describes the placeholders in the paths:
| Placeholder | Value |
|---|---|
| <securityType> | Security type folder from the first table |
| <market> | Lowercase market name, such as usa, cme, oanda, or coinbase |
| <resolution> | tick, second, minute, hour, or daily |
| <ticker> | Lowercase ticker |
| <tickType> | trade, quote, or openinterest |
| <date> | Date of the data in YYYYMMDD format |
| <year> | Year of the data in YYYY format |
| <style> | Option style: american or european |
| <right> | Option right: call or put |
| <strike> | Option strike price multiplied by 10,000 |
| <expiry> | Option expiry date in YYYYMMDD format |
| <contractMonth> | Futures contract month in YYYYMM format |
Price Representation
Prices are in the quote currency of the asset. For example, the value 0.06920 for ETHBTC is the amount of BTC, the quote currency, that buys 1 ETH.
US Equity, Equity Options, and Index Options files store prices multiplied by 10,000. For example, a price of $644.80 is 6448000 in the file. LEAN divides these prices by 10,000 when it reads the files. The files of the other asset classes store the actual prices.
If a security has no activity in a period, the file has no row for that period.
Time Representation
The first column of every row is the time. In tick, second, and minute files, the time is the number of milliseconds since midnight of the file date. Tick times can have a fractional part, such as 43.142. In hour and daily files, the time has the YYYYMMDD HH:mm format, such as 20140606 00:00. The time of a bar is the start of its period.
The times are in the data time zone of the security, which can differ from its exchange time zone. To find the data time zone, look up the dataTimeZone value of the security in the market-hours / market-hours-database.json file. The keys of this file combine the security type, market, and ticker, such as Equity-usa-[*]. The [*] ticker applies to every ticker that has no entry of its own.
LEAN converts the times to the exchange time zone when it reads the files. For more information about time zones, see Data Time Zone.
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
