---
title: "Key Concepts"
description: "There are two techniques to import data into your algorithm. This page explores streaming a file's contents into your algorithm line-by-line."
canonical: https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/key-concepts
markdown: https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/key-concepts.md
language: python
source: https://github.com/QuantConnect/Documentation/tree/master/03%20Writing%20Algorithms/16%20Importing%20Data/02%20Streaming%20Data/01%20Key%20Concepts
modified: 2026-11-02
index: https://www.quantconnect.com/llms.txt
---

# Key Concepts

[Documentation](https://www.quantconnect.com/docs/v2.md) > [Writing Algorithms](https://www.quantconnect.com/docs/v2/writing-algorithms.md) > Importing Data > Streaming Data > Key Concepts

> Code examples are shown in Python. Append `?lang=csharp` to this URL for the C# version of this page.

## Introduction

There are two techniques to import data into your algorithm. You can either manually import the entire file or stream the file line-by-line into your algorithm's `on_data` event. This page explores streaming a file's contents into your algorithm line-by-line. The data you import can be from a remote server or the [Object Store](https://www.quantconnect.com/docs/v2/writing-algorithms/object-store.md).

## Data Formats

Common data formats are `CSV`, `JSON`, `XML`, and `ZIP` but you can use any file type that can be read over the internet. For Excel files, double check the raw data format for parsing in the data reader, since data will be formatted for convenient visualization in Excel application view. To avoid confusion of data format, save the spreadsheet as a `CSV` file and open it in a text editor to confirm the raw data format.

If you import from a remote [file provider](https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/key-concepts.md#02-File-Providers), each request has a one-second overhead, so package your custom data to minimize requests. Bundle dates together where possible to speed up execution. The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution.

## Point-In-Time Tickers

**Tickers** are a string shortcode representation for an asset. Some examples of popular tickers include "AAPL" for Apple Corporation and "IBM" for International Business Machines Corporation. These tickers often change when the company rebrands or they undergo a merger or reverse merger.

The ticker of an asset is not the same as the `Symbol`. `Symbol` objects are permanent and track the underlying entity. When a company rebrands or changes its name, the `Symbol` object remains constant, giving algorithms a way to reliably track assets over time.

Tickers are also often reused by different brokerages. For example Coinbase, a leading US Crypto Brokerage lists the "BTCUSD" ticker for trading. Its competitor, Bitfinex, also lists "BTCUSD". You can trade both tickers with LEAN. `Symbol` objects allow LEAN to identify which market you reference in your algorithms.

To create a `Symbol` object for a point-in-time ticker, call the `generate_equity` method to create the security identifier and then call the `Symbol` constructor. For example, Heliogen, Inc. changed their ticker from ATHN to HLGN on December 31, 2021. To convert the ATHN ticker to the Equity`Symbol`, type:

```python
# Generate an Equity security identifier for a ticker and then create a symbol with it.
ticker = "ATHN"
security_id = SecurityIdentifier.generate_equity(ticker, Market.USA, mapping_resolve_date=datetime(2021, 12, 1))
symbol = Symbol(security_id, ticker)
```

In the preceding code snippet, the `mapping_resolve_date` must be a date when the point-in-time ticker was trading.

For examples of point-in-time tickers in custom data types, refer to the [CSV Format Example](https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/custom-universes/csv-format-example.md#04-Define-Custom-Types) and [JSON Format Example](https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/custom-universes/json-format-example.md#04-Define-Custom-Types).

## Set Data Sources

The `get_source` method in your custom data class instructs LEAN where to find the data.

```python
class MyCustomDataType(PythonData):
    def get_source(self,
         config: SubscriptionDataConfig,
         date: datetime,
         is_live_mode: bool) -> SubscriptionDataSource:

         if is_live_mode:
            return SubscriptionDataSource("https://www.bitstamp.net/api/ticker/", SubscriptionTransportMedium.REST)

        source = f"http://my-ftp-server.com/{config.symbol.value}/{date:%Y%M%d}.csv"
        return SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)

        # Example of loading from the Object Store:
        # return SubscriptionDataSource(Bitstamp.KEY, SubscriptionTransportMedium.OBJECT_STORE)

        # Example of loading a remote zip file:
        # return SubscriptionDataSource(
        #     "https://cdn.quantconnect.com/uploads/multi_csv_zipped_file.zip",
        #     SubscriptionTransportMedium.REMOTE_FILE,
        #     FileFormat.ZIP_ENTRY_NAME
        # )

        # Example of loading a remote zip file and accessing a CSV file inside it:
        # return SubscriptionDataSource(
        #     "https://cdn.quantconnect.com/uploads/multi_csv_zipped_file.zip#csv_file_10.csv",
        #     SubscriptionTransportMedium.REMOTE_FILE,
        #     FileFormat.ZIP_ENTRY_NAME
        # )

        # Example of loading a remote file with Basic authentication headers:
        # import base64
        # user = "username"
        # password = "12345"
        # byte_key = base64.b64encode(f"{user}:{password}".encode("ascii"))
        # authorization_headers = {
        #     "Authorization": f"Basic {byte_key.decode('ascii')}"
        # }
        # return SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE, FileFormat.CSV, authorization_headers)
```

The following table describes the arguments the `get_source` method accepts:

| Argument | Data Type | Description |
| --- | --- | --- |
| `config` | `SubscriptionDataConfig` | The subscription configuration |
| `date` | `datetime` | Date of this source file |
| `is_live_mode` | `bool` | `True` if algorithm is running in live mode |

You can use these arguments to create `SubscriptionDataSource` objects representing different locations and formats. The following table describes the arguments the `SubscriptionDataSource` accepts:

| Argument | Data Type | Description | Default Value |
| --- | --- | --- | --- |
| `source` | `str` | Data source location |  |
| `transport_medium` | `SubscriptionTransportMedium` | The transport medium to be used to retrieve data from the source |  |
| `format` | `FileFormat` | The format of the data within the source | `FileFormat.Csv` |
| `headers` | `dict[str, str]` | The headers to be used for this source. In cloud algorithms, each of the key-value pairs can consist of up to 1,000 characters. | `None` |

The `FileFormat` enumeration has the following members:

*Members of `QuantConnect.Data.FileFormat` (JSON): https://www.quantconnect.com/services/inspector?type=T:QuantConnect.Data.FileFormat&language=python*

The `SubscriptionTransportMedium` enumeration has the following members:

| Member | Description | Example |
| --- | --- | --- |
| `LOCAL_FILE` | The data comes from disk | [Lean.DataSource.BrainSentiment](https://github.com/QuantConnect/Lean.DataSource.BrainSentiment/blob/master/BrainSentimentIndicatorBase.cs#L64) |
| `REMOTE_FILE` | The data is downloaded from a remote source | `Custom Securities Examples` |
| `REST` | The data comes from a rest call that is polled and returns a single line/data point of information | `live_mode` case of [Demonstration Algorithm](https://www.quantconnect.com/docs/v2/writing-algorithms/importing-data/streaming-data/key-concepts.md#99-Demonstration-Algorithm) |
| `OBJECT_STORE` | The data comes from the object store | [Example of Custom Data](https://www.quantconnect.com/docs/v2/writing-algorithms/object-store.md#14-Example-of-Custom-Data) |

## Parse Custom Data

The `reader` method of your custom data class takes one line of data from the source location and parses it into one of your custom objects. You can add as many properties to your custom data objects as you need, but the following table describes the properties you must set. When there is no useable data in a line, the method should return `None`. LEAN repeatedly calls the `reader` method until the date/time advances or it reaches the end of the file.

| Property | Description |
| --- | --- |
| `symbol` | You can set this property to `config.``symbol`. |
| `time` | The time when the data sample starts. |
| `end_time` | The time when the data sample ends and when LEAN should add the sample to a [Slice](https://www.quantconnect.com/docs/v2/writing-algorithms/key-concepts/time-modeling/timeslices.md). |
| `value` | The default data point value (`float`). |

The following table describes the arguments the `reader` method accepts:

| Argument | Data Type | Description |
| --- | --- | --- |
| `config` | `SubscriptionDataConfig` | The subscription configuration |
| `line` | `str` | Content from the requested data source |
| `date` | `datetime` | Date of this source file |
| `is_live_mode` | `bool` | `True` if algorithm is running in live mode |

You can use these arguments to create `BaseData` objects from different sources.

```python
class MyCustomDataType(PythonData):
    def reader(self,
         config: SubscriptionDataConfig,
         line: str,
         date: datetime,
         is_live_mode: bool) -> BaseData:

        if not line.strip():
            return None

        custom = MyCustomDataType()
        custom.symbol = config.symbol

        if is_live_mode:
            data = json.loads(line)
            custom.end_time =  Extensions.convert_from_utc(datetime.utcnow(), config.exchange_time_zone)
            custom.value = data["value"]
            return custom

        if not line[0].isdigit():
            return None

        data = line.split(',')
        custom.end_time = datetime.strptime(data[0], '%Y%m%d') + timedelta(1)
        custom.value = float(data[1])
        return custom
```

## Unsorted Data

By default, LEAN expects the data in chronological order. If the data is unsorted, set the `sort` property of the `SubscriptionDataSource` object to `True`.

```python
class MyCustomDataType(PythonData):
    def get_source(self,
            config: SubscriptionDataConfig,
            date: datetime,
            is_live_mode: bool) -> SubscriptionDataSource:

        source = f"http://my-ftp-server.com/{config.symbol.value}/{date:%Y%M%d}.csv"
        subscription = SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)
        subscription.sort = True
        return subscription
```

LEAN uses `end_time` property of your custom data to sort it.

## Object Store

The Object Store file provider gives you the fastest execution because you don't need to download the files on every execution. To upload the data files into the Object Store, use the [Algorithm Lab](https://www.quantconnect.com/docs/v2/cloud-platform/organizations/object-store.md#03-Upload-Files), [CLI](https://www.quantconnect.com/docs/v2/lean-cli/object-store.md#03-Cloud-Storage), or [Research Environment](https://www.quantconnect.com/docs/v2/research-environment/object-store.md#04-Save-Data). To then pull the data from the Object Store into an algorithm, set the custom data source to read from the file in Object Store and parse the data in the `reader` method.

```python
class MyCustomDataType(PythonData):
    def get_source(self,
            config: SubscriptionDataConfig,
            date: datetime,
            is_live_mode: bool) -> SubscriptionDataSource:

        return SubscriptionDataSource("<your_csv_key>", SubscriptionTransportMedium.OBJECT_STORE, FileFormat.CSV)
        # return new SubscriptionDataSource("<your_json_key>", SubscriptionTransportMedium.ObjectStore, FileFormat.UNFOLDING_COLLECTION);
```

## Set Properties

To set the [Symbol Properties](https://www.quantconnect.com/docs/v2/writing-algorithms/securities/key-concepts.md#06-Symbol-Properties) of the custom data, provide a `SymbolProperties` object when you subscribe to the dataset. The ticker you pass to the `SymbolProperties` constructor and the `add_data` method must be the same.

To set the [Exchange Hours](https://www.quantconnect.com/docs/v2/writing-algorithms/securities/market-hours.md) of the custom data, provide a `SecurityExchangeHours` object when you subscribe to the dataset. The default hours are for the market to be open 24/7.

```python
ticker = "ABC"
properties = SymbolProperties("description", "USD", 1, 0.01, 0.01, ticker)
hours = MarketHoursDatabase.get_entry(Market.USA, "SPY", SecurityType.EQUITY).exchange_hours
self.add_data(CustomData, ticker, properties, hours)
```

## Live Trading Considerations

In live trading, we pass custom data to your algorithm as soon as it arrives. The time it arrives may not align with the time of other [slices](https://www.quantconnect.com/docs/v2/writing-algorithms/key-concepts/time-modeling/timeslices.md). Design your algorithm to handle unsychronized data so that you don't run into issues.

## Demonstration Algorithm

The following example algorithm implements a custom data source for the Bitstamp API.

```python
class CustomDataBitstampAlgorithm(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2020, 9, 1)
        self.set_end_date(2020, 12, 31)
        self.set_cash(100000)
        # Define the symbol and "type" of our generic data:
        self._custom_data_symbol = self.add_data(Bitstamp, "BTC").symbol
        # Get some historical data.
        history = self.history(
            Bitstamp, self._custom_data_symbol, 200, Resolution.DAILY
        )

    def on_data(self, slice):
        # Get the data of the current day.
        data = slice.get(self._custom_data_symbol)
        if not data:
            return
        self.plot(self._custom_data_symbol, 'Price', data.close)

class Bitstamp(PythonData):

    def get_source(self, config, date, is_live_mode):
        if is_live_mode:
            return SubscriptionDataSource(
                'https://www.bitstamp.net/api/ticker/',
                SubscriptionTransportMedium.REST
            )
        return SubscriptionDataSource(
            "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv",
            SubscriptionTransportMedium.REMOTE_FILE
        )

    def reader(self, config, line, date, is_live_mode):
        if not line.strip():
            return None
        coin = Bitstamp()
        coin.symbol = config.symbol
        # In live trading, parse the JSON file.
        if is_live_mode:
            # Example Line Format:
            # {"high": "441.00", "last": "421.86", "timestamp": "1411606877", "bid": "421.96", "vwap": "428.58", "volume": "14120.40683975", "low": "418.83", "ask": "421.99"}
            live_btc = json.loads(line)
            # If value is zero, return None
            coin.value = float(live_btc["last"])
            if coin.value == 0:
                return None
            coin.end_time =  Extensions.convert_from_utc(
                datetime.utcnow(), config.exchange_time_zone
            )
            coin.time = coin.end_time - timedelta(1)
            coin["Open"] = float(live_btc["open"])
            coin["High"] = float(live_btc["high"])
            coin["Low"] = float(live_btc["low"])
            coin["Close"] = coin.value
            coin["Ask"] = float(live_btc["ask"])
            coin["Bid"] = float(live_btc["bid"])
            coin["VolumeBTC"] = float(live_btc["volume"])
            coin["WeightedPrice"] = float(live_btc["vwap"])
            return coin

        # In backtests, parse the CSV file.
        # Example Line Format:
        # Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        # 2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if not line[0].isdigit():
            return None
        data = line.split(',')
        # If value is zero, return None
        coin.value = float(data[4])
        if coin.value == 0:
            return None
        coin.time = datetime.strptime(data[0], "%Y-%m-%d")
        coin.end_time = coin.time + timedelta(1)
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = coin.value
        coin["VolumeBTC"] = float(data[5])
        coin["VolumeUSD"] = float(data[6])
        coin["WeightedPrice"] = float(data[7])
        return coin
```