Writing Algorithms

To get all of the keys and values in the Object Store, iterate through the ObjectStoreobject_store property.

foreach (var kvp in ObjectStore)
{
    var key = kvp.Key;
    var value = kvp.Value;
}

for kvp in self.object_store:
    key = kvp.key
    value = kvp.value

To iterate through just the keys in the Object Store, iterate through the Keyskeys property.

foreach (var key in ObjectStore.Keys)
{
    continue;
}

for key in self.object_store.keys:
    continue

You need some data to store data in the Object Store.

Follow these steps to create some sample data:

1. Create a dictionary.

var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };

2. Create a string.

var stringSample = "My string";

string_sample = "My string"

3. Create a Bytes object.

var bytesSample = Encoding.UTF8.GetBytes("My String");

bytes_sample = str.encode("My String")

4. Convert the dictionary to an XML-formatted object.

var xmlSample = new XElement("sample",
    dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value)));
Log(xmlSample.ToString());

The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment. To avoid slowing down your backtests, save data once in the OnEndOfAlgorithmon_end_of_algorithm event handler. In live trading, you can save data more frequently like at the end of a Traintrain method or after universe selection.

If you run algorithms in QuantConnect Cloud, you need storage create permissions to save data in the Object Store.

If you don't have data to store, create some sample data.

You can save the following types of objects in the Object Store:

• Bytes objects
• string objects
• JSON objects
• XML-formatted objects

You can save Bytes and string objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

Strings

To save a string object, call the Savesave or SaveStringsave_string method.

var saveSuccessful = ObjectStore.Save($"{ProjectId}/stringKey", stringSample);

save_successful = self.object_store.save(f"{self.project_id}/string_key", string_sample)

JSON

To save a JSON object, call the SaveJson<T> method. This method helps to serialize the data into JSON format.

var saveSuccessful = ObjectStore.SaveJson<Dictionary<string, int>>($"{ProjectId}/jsonKey", dictSample);

XML

To save an XML-formatted object, call the SaveXml<T> method.

var saveSuccessful = ObjectStore.SaveXml<XElement>($"{ProjectId}/xmlKey", xmlSample);

Bytes

To save a Bytes object (for example, zipped data), call the SaveBytessave_bytes method.

var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey", bytesSample)

var zippedDataSample = Compression.ZipBytes(Encoding.UTF8.GetBytes(stringSample), "data");
var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey.zip", zippedDataSample);

save_successful = self.object_store.save_bytes(f"{self.project_id}/bytes_key", bytes_sample)

zipped_data_sample = Compression.zip_bytes(bytes(string_sample, "utf-8"), "data")
zip_save_successful = self.object_store.save_bytes(f"{self.project_id}/bytesKey.zip", zipped_data_sample)

Read data from the Object Store to import algorithm variables between deployments, import data from the Research Environment, or load trained machine learning models. To read data from the Object Store, you need to provide the key you used to store the object.

You can load the following types of objects from the Object Store:

• Bytes objects
• string objects
• JSON objects
• XML-formatted objects

You can load Bytes and string objects from the Object Store.

Before you read data from the Object Store, check if the key exists.

if (ObjectStore.ContainsKey(key))
{
    // Read data
}

if self.object_store.contains_key(key):
    # Read data

Strings

To read a string object, call the Readread or ReadStringread_string method.

var stringData = ObjectStore.Read($"{ProjectId}/stringKey");

string_data = self.object_store.read(f"{self.project_id}/string_key")

JSON

To read a JSON object, call the ReadJson<T> method.

var jsonData = ObjectStore.ReadJson<Dictionary<string, int>>($"{ProjectId}/jsonKey");

XML

To read an XML-formatted object, call the ReadXml<T> method.

var xmlData = ObjectStore.ReadXml<XElement>($"{ProjectId}/xmlKey");

If you created the XML object from a dictionary, reconstruct the dictionary.

var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));

Bytes

To read a Bytes object, call the ReadBytesread_bytes method.

var bytesData = ObjectStore.ReadBytes($"{ProjectId}/bytesKey");

byte_data = self.object_store.read_bytes(f"{self.project_id}/bytes_key")

Delete objects in the Object Store to remove objects that you no longer need. If you run algorithms in QuantConnect Cloud, you need storage delete permissions to delete data from the Object Store.

To delete objects from the Object Store, call the Deletedelete method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.

if (ObjectStore.ContainsKey(key))
{
    ObjectStore.Delete(key);
}

if self.object_store.contains_key(key):
    self.object_store.delete(key)

To delete all of the content in the Object Store, iterate through all the stored data.

foreach (var kvp in ObjectStore)
{
    ObjectStore.Delete(kvp.Key);
}

for kvp in self.object_store:
    self.object_store.delete(kvp.key)

When you write to or read from the Object Store, the algorithm caches the data. The cache speeds up the algorithm execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:

1. You open project A and save data under the key 123.
2. You open project B and save new data under the same key 123.
3. In project A, you read the Object Store data under the key 123, expecting the data from project B, but you get the original data you saved in step #1 instead.

You get the data from step 1 instead of step 2 because the cache contains the data from step 1.

To clear the cache, call the Clear method.

ObjectStore.Clear();

self.object_store.clear()

To get the file path for a specific key in the Object Store, call the GetFilePathget_file_path method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.

var filePath = ObjectStore.GetFilePath(key);

file_path = self.object_store.get_file_path(key)

If you run algorithms locally, you can store as much data as your hardware will allow. If you run algorithms in QuantConnect Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.

Follow these steps to create a DataFrame, save it into the Object Store, and load it from the Object Store:

1. Get some historical data.

var spy = AddEquity("SPY").Symbol;
var history = History(Securities.Keys, 360, Resolution.Daily);

spy = self.add_equity("SPY").symbol
df = self.history(self.securities.keys, 360, Resolution.DAILY)

2. Create a DataFrame.

using Microsoft.Data.Analysis; // 

var columns = new DataFrameColumn[] {
    new DateTimeDataFrameColumn("Time", history.Select(x => (DateTime)x[spy].EndTime)),
    new DecimalDataFrameColumn("SPY Open", history.Select(x => (decimal)x[spy].Open)),
    new DecimalDataFrameColumn("SPY High", history.Select(x => (decimal)x[spy].High)),
    new DecimalDataFrameColumn("SPY Low", history.Select(x => (decimal)x[spy].Low)),
    new DecimalDataFrameColumn("SPY Close", history.Select(x => (decimal)x[spy].Close))
};
var df = new DataFrame(columns);

3. Get the file path for a specific key in the Object Store.

var filePath = ObjectStore.GetFilePath("df_to_csv");

file_path = self.object_store.get_file_path("df_to_csv")

4. Call the SaveCsvto_csv method to save the DataFrame in the Object Store as a CSV file.

DataFrame.SaveCsv(df, filePath);    // File size: 26520 bytes

df.to_csv(file_path)   # File size: 32721 bytes

5. Call the LoadCsvread_csv method to load the CSV file from the Object Store.

var reread = DataFrame.LoadCsv(filePath);

reread = pd.read_csv(file_path)

pandas supports saving and loading DataFrame objects in the following additional formats:

• XML

file_path = self.object_store.get_file_path("df_to_xml")
df.to_xml(file_path)   # File size: 87816 bytes
reread = pd.read_xml(file_path)

• JSON

file_path = self.object_store.get_file_path("df_to_json")
df.to_json(file_path)   # File size: 125250 bytes
reread = pd.read_json(file_path)

• Parquet

file_path = self.object_store.get_file_path("df_to_parquet")
df.to_parquet(file_path)   # File size: 23996 bytes
reread = pd.read_parquet(file_path)

• Pickle

file_path = self.object_store.get_file_path("df_to_pickle")
df.to_pickle(file_path)   # File size: 19868 bytes
reread = pd.read_pickle(file_path)

You can use the Object Store to plot data from your backtests and live algorithm in the Research Environment. The following example demonstrates how to plot a Simple Moving Average indicator that's generated during a backtest.

1. Create a algorithm, add a data subscription, and add a simple moving average indicator.

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private SimpleMovingAverage _sma;
    private string _content;

    public override void Initialize()
    {
        AddEquity("SPY", Resolution.Minute);
        _sma = SMA("SPY", 22);
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def initialize(self):
        self.add_equity("SPY")
    
        self.content = ''
        self._sma = self.sma("SPY", 22)

The algorithm will save _contentself.content to the Object Store.

2. Save the indicator data as string in _contentself.content.

public override void OnData(Slice data)
{
    _content += $"{_sma.Current.EndTime},{_sma}\n";
}

def on_data(self, data: Slice):
    self.plot('SMA', 'Value', self.sma.current.value)
    self.content += f'{self.sma.current.end_time},{self.sma.current.value}\n'

3. In the OnEndOfAlgorithm method, save the indicator data to the Object Store.

public override void OnEndOfAlgorithm()
{
    ObjectStore.Save("sma_values_csharp", _content);
}

def on_end_of_algorithm(self):
    self.object_store.save('sma_values_python', self.content)

4. Open the Research Environment and create a QuantBook.

// Execute the following command in first
#load "../Initialize.csx"

// Create a QuantBook object
#load "../QuantConnect.csx"
using QuantConnect;
using QuantConnect.Research;

var qb = new QuantBook();

qb = QuantBook()

5. Read the indicator data from the Object Store.

var content = qb.ObjectStore.Read("sma_values_csharp");

content = qb.object_store.read("sma_values_python")

The key you provide must be the same key you used to save the object.

6. Convert the data to a pandas object and create a chart.

data = {}
for line in content.split('\n'):
    csv = line.split(',')
    if len(csv) > 1:
        data[csv[0]] = float(csv[1])

series = pd.Series(data, index=data.keys())
series.plot()

7. Import the Plotly.NET and Plotly.NET.LayoutObjects packages.

#r "../Plotly.NET.dll"
using Plotly.NET;
using Plotly.NET.LayoutObjects;

8. Create the Layout object and set the title, xaxis, and yaxis properties.

var layout = new Layout();
layout.SetValue("title", Title.init("SMA"));

var xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
layout.SetValue("xaxis", xAxis);

var yAxis = new LinearAxis();
yAxis.SetValue("title", "SMA");
layout.SetValue("yaxis", yAxis);

9. Convert the data to a list of DateTime objects for the chart x-axis and a list of decimal objects for the chart y-axis, then create a Chart2D.Chart.Line object with the data.

var index = new List<DateTimee>();
var values = new List<decimal>();

foreach (var line in content.Split('\n'))
{
    var csv = line.Split(',');
    if (csv.Length > 1)
    {
        index.Add(Parse.DateTime(csv[0]));
        values.Add(decimal.Parse(csv[1]));
    }
}

var chart = Chart2D.Chart.Linee<DateTime, decimal, stringe>(index, values);

10. Apply the layout to the Line object and create the HTML object.

chart.WithLayout(layout);
var result = HTML(GenericChart.toChartHTML(chart));

Follow these steps to use the Object Store as a data source for custom data:

1. Create a custom data class that defines a storage key and implements the GetSourceget_source method.

public class Bitstamp : TradeBar
{
    public static string KEY = "bitstampusd.csv";
            
    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
    {
        return new SubscriptionDataSource(KEY, SubscriptionTransportMedium.ObjectStore);
    }
}    

class Bitstamp(PythonData):
    KEY = 'bitstampusd.csv'
    def get_source(self, config, date, isLiveMode):
        return SubscriptionDataSource(Bitstamp.KEY, SubscriptionTransportMedium.OBJECT_STORE)

2. Create an algorithm that downloads data from an external source and saves it to the Object Store.

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        if (!ObjectStore.ContainsKey(Bitstamp.KEY))
        {
            var url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv";
            var content = Download(url);
            ObjectStore.Save(Bitstamp.KEY, content);
        }   
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        if not self.object_store.contains_key(Bitstamp.KEY):
            url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv"
            content = self.download(url)
            self.object_store.save(Bitstamp.KEY, content)

3. Call the AddDataadd_data method to subscribe to the custom type.

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    private Symbol _customDataSymbol;
    public override void Initialize()
    {
        _customDataSymbol = AddData<Bitstamp>("BTC").Symbol;  
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        self.custom_data_symbol = self.add_data(Bitstamp, "BTC").symbol

4. Implement the Reader method for the custom data class.

public class Bitstamp : TradeBar
{
    public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
    {
        //Example Line Format:
        //Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        //2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if (!char.IsDigit(line.Trim()[0]))
        {
            return null;
        }

        var coin = new Bitstamp() {Symbol = config.Symbol};

        var data = line.Split(',');
        coin.Value = data[4].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        if (coin.Value == 0)
        {
            return null;
        }

        coin.Time = DateTime.Parse(data[0], CultureInfo.InvariantCulture);
        coin.EndTime = coin.Time.AddDays(1);
        coin.Open = data[1].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.High = data[2].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Low = data[3].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.VolumeBTC = data[5].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Volume = data[6].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.WeightedPrice = data[7].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Close = coin.Value;
        return coin;
    }
}

class Bitstamp(PythonData):
    def reader(self, config, line, date, isLiveMode):
        # Example Line Format:
        # Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        # 2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if not line[0].isdigit():
            return None

        coin = Bitstamp()
        coin.symbol = config.symbol
        data = line.split(',')

        # If value is zero, return None
        coin.value = float(data[4])
        if coin.value == 0:
            return None

        coin.time = datetime.strptime(data[0], "%Y-%m-%d")
        coin.end_time = coin.time + timedelta(1)
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = coin.value
        coin["VolumeBTC"] = float(data[5])
        coin["VolumeUSD"] = float(data[6])
        coin["WeightedPrice"] = float(data[7])
        return coin

5. To confirm your algorithm is receiving the custom data, request some historical data, then log and plot the data from the Slice object.

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        var history = History(_customDataSymbol, 200, Resolution.Daily);
        Debug($"We got {history.Count()} items from historical data request of {_customDataSymbol}.");
    }
    public void OnData(Bitstamp data)
    {
        Log($"{data.EndTime}: Close: {data.Close}");
        Plot(_customDataSymbol, "Price", data.Close);
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        history = self.history(Bitstamp, self.custom_data_symbol, 200, Resolution.DAILY)
        self.debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")
    def on_data(self, slice):
        data = slice.get(Bitstamp).get( self.custom_data_symbol)
        if not data:
            return

        self.log(f'{data.end_time}: Close: {data.close}')
        self.plot(self.custom_data_symbol, 'Price', data.close)

The following algorithm provides a full example of a sourcing custom data from the Object Store:

Follow these steps to use the Object Store to preserve the algorithm state across live deployments:

1. Create an algorithm that defines a storage key and adds insights to the Insight Manager.

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private string _insightKey;
    public override void Initialize()
    {
        _insightKey = $"{ProjectId}/insights";
        SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
        SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));    
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def initialize(self):
        self.insight_key = f"{self.project_id}/insights"
        self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
        self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))

2. At the top of the algorithm file, add the following imports:

from Newtonsoft.Json import JsonConvert
from System.Collections.Generic import List

Insight objects are a C# objects, so you need the preceding C# libraries to serialize and deserialize them.

3. In the OnEndOfAlgorithm event handler of the algorithm, get the Insight objects and save them in the Object Store as a JSON object.

public override void OnEndOfAlgorithm()
{
    var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
    ObjectStore.SaveJson(_insightKey, insights);
}

def on_end_of_algorithm(self):
    insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
    content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
    self.object_store.save(self.insight_key, f'[{content}]')

4. At the bottom of the Initializeinitialize method, read the Insight objects from the Object Store and add them to the Insight Manager.

if (ObjectStore.ContainsKey(_insightKey))
{
    var insights = ObjectStore.ReadJson<List<Insight>>(_insightKey);
    Insights.AddRange(insights);
}

if self.object_store.contains_key(self.insight_key):
    insights = self.object_store.read_json[List[Insight]](self.insight_key)
    self.insights.add_range(insights)

The following algorithm provides a full example of preserving the Insight state between deployments:

If you update the Object Store from outside of your live trading node and then try to read the new content from inside your live trading algorithm, you may not get the new content because the Object Store caches data to speed up execution. To ensure you get the latest value for a specific key in the Object Store, clear the cache and then read the data.

ObjectStore.Clear();
if (ObjectStore.ContainsKey(key))
{
    var value = ObjectStore.Read(key);
}

self.object_store.clear()
if self.object_store.contains_key(key):
    value = self.object_store.read(key)