Writing Algorithms

To get all of the keys and values in the Object Store, iterate through the ObjectStore object.

foreach (var kvp in ObjectStore)
{
    var key = kvp.Key;
    var value = kvp.Value;
}

for kvp in self.ObjectStore:
    key = kvp.Key
    value = kvp.Value

To iterate through just the keys in the Object Store, iterate through the Keys property.

foreach (var key in ObjectStore.Keys)
{
    continue;
}

for key in self.ObjectStore.Keys:
    continue

You need some data to store data in the Object Store.

Follow these steps to create some sample data:

1. Create a dictionary.

var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };

2. Create a string.

var stringSample = "My string";

string_sample = "My string"

3. Create a Bytes object.

var bytesSample = Encoding.UTF8.GetBytes("My String");

bytes_sample = str.encode("My String")

4. Convert the dictionary to an XML-formatted object.

var xmlSample = new XElement("sample",
    dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value)));
Log(xmlSample.ToString());

The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment. To avoid slowing down your backtests, save data once in the OnEndOfAlgorithm event handler. In live trading, you can save data more frequently like at the end of a Train method or after universe selection.

If you run algorithms in QuantConnect Cloud, you need storage create permissions to save data in the Object Store.

If you don't have data to store, create some sample data.

You can save the following types of objects in the Object Store:

• Bytes objects
• string objects
• JSON objects
• XML-formatted objects

You can save Bytes and string objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

Strings

To save a string object, call the Save or SaveString method.

var saveSuccessful = ObjectStore.Save($"{ProjectId}/stringKey", stringSample);

save_successful = self.ObjectStore.Save(f"{self.ProjectId}/string_key", string_sample)

JSON

To save a JSON object, call the SaveJson<T> method. This method helps to serialize the data into JSON format.

var saveSuccessful = ObjectStore.SaveJson<Dictionary<string, int>>($"{ProjectId}/jsonKey", dictSample);

XML

To save an XML-formatted object, call the SaveXml<T> method.

var saveSuccessful = ObjectStore.SaveXml<XElement>($"{ProjectId}/xmlKey", xmlSample);

Bytes

To save a Bytes object (for example, zipped data), call the SaveBytes method.

var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey", bytesSample)

var zippedDataSample = Compression.ZipBytes(Encoding.UTF8.GetBytes(stringSample), "data");
var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey.zip", zippedDataSample);

save_successful = self.ObjectStore.SaveBytes(f"{self.ProjectId}/bytes_key", bytes_sample)

zipped_data_sample = Compression.ZipBytes(bytes(string_sample, "utf-8"), "data")
zip_save_successful = self.ObjectStore.SaveBytes($"{self.ProjectId}/bytesKey.zip", zipped_data_sample)

Read data from the Object Store to import algorithm variables between deployments, import data from the Research Environment, or load trained machine learning models. To read data from the Object Store, you need to provide the key you used to store the object.

You can load the following types of objects from the Object Store:

• Bytes objects
• string objects
• JSON objects
• XML-formatted objects

You can load Bytes and string objects from the Object Store.

Before you read data from the Object Store, check if the key exists.

if (ObjectStore.ContainsKey(key))
{
    // Read data
}

if self.ObjectStore.ContainsKey(key):
    # Read data

Strings

To read a string object, call the Read or ReadString method.

var stringData = ObjectStore.Read($"{ProjectId}/stringKey");

string_data = self.ObjectStore.Read(f"{self.ProjectId}/string_key")

JSON

To read a JSON object, call the ReadJson<T> method.

var jsonData = ObjectStore.ReadJson<Dictionary<string, int>>($"{ProjectId}/jsonKey");

XML

To read an XML-formatted object, call the ReadXml<T> method.

var xmlData = ObjectStore.ReadXml<XElement>($"{ProjectId}/xmlKey");

If you created the XML object from a dictionary, reconstruct the dictionary.

var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));

Bytes

To read a Bytes object, call the ReadBytes method.

var bytesData = ObjectStore.ReadBytes($"{ProjectId}/bytesKey");

byte_data = self.ObjectStore.ReadBytes(f"{self.ProjectId}/bytes_key")

Delete objects in the Object Store to remove objects that you no longer need. If you run algorithms in QuantConnect Cloud, you need storage delete permissions to delete data from the Object Store.

To delete objects from the Object Store, call the Delete method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.

if (ObjectStore.ContainsKey(key))
{
    ObjectStore.Delete(key);
}

if self.ObjectStore.ContainsKey(key):
    self.ObjectStore.Delete(key)

To delete all of the content in the Object Store, iterate through all the stored data.

foreach (var kvp in ObjectStore)
{
    ObjectStore.Delete(kvp.Key);
}

for kvp in self.ObjectStore:
    self.ObjectStore.Delete(kvp.Key)

When you write to or read from the Object Store, the algorithm caches the data. The cache speeds up the algorithm execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:

1. You open project A and save data under the key 123.
2. You open project B and save new data under the same key 123.
3. In project A, you read the Object Store data under the key 123, expecting the data from project B, but you get the original data you saved in step #1 instead.

You get the data from step 1 instead of step 2 because the cache contains the data from step 1.

To clear the cache, call the Clear method.

ObjectStore.Clear();

self.ObjectStore.Clear()

To get the file path for a specific key in the Object Store, call the GetFilePath method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.

var filePath = ObjectStore.GetFilePath(key);

file_path = self.ObjectStore.GetFilePath(key)

If you run algorithms locally, you can store as much data as your hardware will allow. If you run algorithms in QuantConnect Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.

Follow these steps to create a DataFrame, save it into the Object Store, and load it from the Object Store:

1. Get some historical data.

var spy = AddEquity("SPY").Symbol;
var history = History(Securities.Keys, 360, Resolution.Daily);

spy = self.AddEquity("SPY").Symbol
df = self.History(self.Securities.Keys, 360, Resolution.Daily)

2. Create a DataFrame.

using Microsoft.Data.Analysis; // 

var columns = new DataFrameColumn[] {
    new DateTimeDataFrameColumn("Time", history.Select(x => (DateTime)x[spy].EndTime)),
    new DecimalDataFrameColumn("SPY Open", history.Select(x => (decimal)x[spy].Open)),
    new DecimalDataFrameColumn("SPY High", history.Select(x => (decimal)x[spy].High)),
    new DecimalDataFrameColumn("SPY Low", history.Select(x => (decimal)x[spy].Low)),
    new DecimalDataFrameColumn("SPY Close", history.Select(x => (decimal)x[spy].Close))
};
var df = new DataFrame(columns);

3. Get the file path for a specific key in the Object Store.

var filePath = ObjectStore.GetFilePath("df_to_csv");

file_path = self.ObjectStore.GetFilePath("df_to_csv")

4. Call the SaveCsvto_csv method to save the DataFrame in the Object Store as a CSV file.

DataFrame.SaveCsv(df, filePath);    // File size: 26520 bytes

df.to_csv(file_path)   # File size: 32721 bytes

5. Call the LoadCsvread_csv method to load the CSV file from the Object Store.

var reread = DataFrame.LoadCsv(filePath);

reread = pd.read_csv(file_path)

pandas supports saving and loading DataFrame objects in the following additional formats:

• XML

file_path = self.ObjectStore.GetFilePath("df_to_xml")
df.to_xml(file_path)   # File size: 87816 bytes
reread = pd.read_xml(file_path)

• JSON

file_path = self.ObjectStore.GetFilePath("df_to_json")
df.to_json(file_path)   # File size: 125250 bytes
reread = pd.read_json(file_path)

• Parquet

file_path = self.ObjectStore.GetFilePath("df_to_parquet")
df.to_parquet(file_path)   # File size: 23996 bytes
reread = pd.read_parquet(file_path)

• Pickle

file_path = self.ObjectStore.GetFilePath("df_to_pickle")
df.to_pickle(file_path)   # File size: 19868 bytes
reread = pd.read_pickle(file_path)

You can use the Object Store to plot data from your backtests and live algorithm in the Research Environment. In the following example, you will learn how to plot the Simple Moving Average indicator generated in a backtest.

1. Create a algorithm, add a data subscription and a Simple Moving Average indicator.

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private SimpleMovingAverage _sma;
    private string _content;

    public override void Initialize()
    {
        AddEquity("SPY", Resolution.Minute);
        _sma = SMA("SPY", 22);
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def Initialize(self):
        self.AddEquity("SPY")
    
        self.content = ''
        self.sma = self.SMA("SPY", 22)

The algorithm will save _contentself.content to the Object Store.

2. Save indicator data as string in _contentself.content.

public override void OnData(Slice data)
{
    _content += $"{_sma.Current.EndTime},{_sma}\n";
}

def OnData(self, data: Slice):
    self.Plot('SMA', 'Value', self.sma.Current.Value)
    self.content += f'{self.sma.Current.EndTime},{self.sma.Current.Value}\n'

3. To store the collected data, call the Save method with a key.

public override void OnEndOfAlgorithm()
{
    ObjectStore.Save("sma_values_csharp", _content);
}

def OnEndOfAlgorithm(self):
    self.ObjectStore.Save('sma_values_python', self.content)

4. Open the Research Environment, and create a QuantBook.

// Execute the following command in first
#load "../Initialize.csx"

// Create a QuantBook object
#load "../QuantConnect.csx"
using QuantConnect;
using QuantConnect.Research;

var qb = new QuantBook();

qb = QuantBook()

5. To read data from the Object Store, call the Read method. You need to provide the key you used to store the object.

var content = qb.ObjectStore.Read("sma_values_csharp");

content = qb.ObjectStore.Read("sma_values_python")

6. Convert the data to a pandas object, and create a chart.

data = {}
for line in content.split('\n'):
    csv = line.split(',')
    if len(csv) > 1:
        data[csv[0]] = float(csv[1])

series = pd.Series(data, index=data.keys())
series.plot()

7. Import the Plotly.NET and Plotly.NET.LayoutObjects packages.

#r "../Plotly.NET.dll"
using Plotly.NET;
using Plotly.NET.LayoutObjects;

8. Create the Layout object, and set the title, xaxis, and yaxis properties.

var layout = new Layout();
layout.SetValue("title", Title.init("SMA"));

var xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
layout.SetValue("xaxis", xAxis);

var yAxis = new LinearAxis();
yAxis.SetValue("title", "SMA");
layout.SetValue("yaxis", yAxis);

9. Convert the data to a list of DateTime objects for the chart x-axis and a list of decimal objects for the chart y-axis. Create a Chart2D.Chart.Line object with the data.

var index = new List<DateTimee>();
var values = new List<decimal>();

foreach (var line in content.Split('\n'))
{
    var csv = line.Split(',');
    if (csv.Length > 1)
    {
        index.Add(Parse.DateTime(csv[0]));
        values.Add(decimal.Parse(csv[1]));
    }
}

var chart = Chart2D.Chart.Linee<DateTime, decimal, stringe>(index, values);

10. Apply the layout to the Line object, and create the HTML object.

chart.WithLayout(layout);
var result = HTML(GenericChart.toChartHTML(chart));

Follow these steps to use the Object Store to preserve the algorithm state across live deployments:

1. Create an algorithm that defines a storage key and adds insights to the Insight Manager.

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private string _insightKey;
    public override void Initialize()
    {
        _insightKey = $"{ProjectId}/insights";
        SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
        SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));    
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def Initialize(self):
        self.insight_key = f"{self.ProjectId}/insights"
        self.SetUniverseSelection(ManualUniverseSelectionModel([ Symbol.Create("SPY", SecurityType.Equity, Market.USA) ]))
        self.SetAlpha(ConstantAlphaModel(InsightType.Price, InsightDirection.Up, timedelta(5), 0.025, None))

2. At the top of the algorithm file, add the following imports:

from Newtonsoft.Json import JsonConvert
from System.Collections.Generic import List

Insight objects are a C# objects, so you need the preceding C# libraries to serialize and deserialize them.

3. In the OnEndOfAlgorithm event handler of the algorithm, get the Insight objects and save them in the Object Store as a JSON object.

public override void OnEndOfAlgorithm()
{
    var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
    ObjectStore.SaveJson(_insightKey, insights);
}

def OnEndOfAlgorithm(self):
    insights = self.Insights.GetInsights(lambda x: x.IsActive(self.UtcTime))
    content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
    self.ObjectStore.Save(self.insight_key, f'[{content}]')

4. At the bottom of the Initialize method, read the Insight objects from the Object Store and add them to the Insight Manager.

if (ObjectStore.ContainsKey(_insightKey))
{
    var insights = ObjectStore.ReadJson<List<Insight>>(_insightKey);
    Insights.AddRange(insights);
}

if self.ObjectStore.ContainsKey(self.insight_key):
    insights = self.ObjectStore.ReadJson[List[Insight]](self.insight_key)
    self.Insights.AddRange(insights)

The following algorithm provides a full example of preserving the Insight state between deployments:

If you update the Object Store from outside of your live trading node and then try to read the new content from inside your live trading algorithm, you may not get the new content because the Object Store caches data to speed up execution. To ensure you get the latest value for a specific key in the Object Store, clear the cache and then read the data.

ObjectStore.Clear();
if (ObjectStore.ContainsKey(key))
{
    var value = ObjectStore.Read(key);
}

self.ObjectStore.Clear()
if self.ObjectStore.ContainsKey(key):
    value = self.ObjectStore.Read(key)