Contents
Writing Algorithms
Object Store
Introduction
The Object Store is a file system that you can use in your algorithms to save, read, and delete data. The Object Store is organization-specific, so you can save or read data from the same Object Store in all of your organization's projects. The Object Store works like a key-value storage system where you can store regular strings, JSON encoded strings, XML encoded strings, and bytes. You can access the data you store in the Object Store from backtests, the Research Environment, and live algorithms.
When you deploy live algorithms, the state of the Object Store is copied, but it never refreshes. Therefore, if you save data in the Object Store in a live algorithm, you can access the data from the live algorithm, backtests, and the Research Environment. However, if you save content into the Object Store from the Research Environment or a backtest after you deploy a live algorithm, you can't access the new content from the live algorithm.
Get All Stored Data
To get all of the keys and values in the Object Store, iterate through the ObjectStore
object.
foreach (var kvp in ObjectStore) { var key = kvp.Key; var value = kvp.Value; }
for kvp in self.ObjectStore: key = kvp.Key value = kvp.Value
To iterate through just the keys in the Object Store, iterate through the Keys
property.
foreach (var key in ObjectStore.Keys) { continue; }
for key in self.ObjectStore.Keys: continue
Create Sample Data
You need some data to store data in the Object Store.
Follow these steps to create some sample data:
- Create a dictionary.
- Create a
string
. - Create a
Bytes
object. - Convert the dictionary to an
XML
-formatted object.
var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };
var stringSample = "My string";
string_sample = "My string"
var bytesSample = Encoding.UTF8.GetBytes("My String");
bytes_sample = str.encode("My String")
var xmlSample = new XElement("sample", dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value))); Log(xmlSample.ToString());

Get File Path
To get the file path for a specific key in the Object Store, call the GetFilePath
method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.
var filePath = ObjectStore.GetFilePath(key);
file_path = self.ObjectStore.GetFilePath(key)
Save Data
The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment. To avoid slowing down your backtests, save data once in the OnEndOfAlgorithm
event handler. In live trading, you can save data more frequently like at the end of a Train
method or after universe selection.
If you run algorithms in QuantConnect Cloud, you need storage create permissions to save data in the Object Store.
If you don't have data to store, create some sample data.
You can save the following types of objects in the Object Store:
Bytes
objectsstring
objects- JSON objects
- XML-formatted objects
To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.
You can save Bytes
and string
objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.
Bytes
To save a Bytes
object, call the SaveBytes
method.
var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey", bytesSample)
save_successful = self.ObjectStore.SaveBytes(f"{self.ProjectId}/bytes_key", bytes_sample)
Strings
To save a string
object, call the Save
or SaveString
method.
var saveSuccessful = ObjectStore.Save($"{ProjectId}/stringKey", stringSample);
save_successful = self.ObjectStore.Save(f"{self.ProjectId}/string_key", string_sample)
JSON
To save a JSON object, call the SaveJson<T>
method. This method helps to serialize the data into JSON format.
var saveSuccessful = ObjectStore.SaveJson<Dictionary<string, int>>($"{ProjectId}/jsonKey", dictSample);
XML
To save an XML-formatted object, call the SaveXml<T>
method.
var saveSuccessful = ObjectStore.SaveXml<XElement>($"{ProjectId}/xmlKey", xmlSample);
Read Data
Read data from the Object Store to import algorithm variables between deployments, import data from the Research Environment, or load trained machine learning models. To read data from the Object Store, you need to provide the key you used to store the object.
You can load the following types of objects from the Object Store:
Bytes
objectsstring
objects- JSON objects
- XML-formatted objects
You can load Bytes
and string
objects from the Object Store.
Before you read data from the Object Store, check if the key exists.
if (ObjectStore.ContainsKey(key)) { // Read data }
if self.ObjectStore.ContainsKey(key): # Read data
Bytes
To read a Bytes
object, call the ReadBytes
method.
var bytesData = ObjectStore.ReadBytes($"{ProjectId}/bytesKey");
byte_data = self.ObjectStore.ReadBytes(f"{self.ProjectId}/bytes_key")
Strings
To read a string
object, call the Read
or ReadString
method.
var stringData = ObjectStore.Read($"{ProjectId}/stringKey");
string_data = self.ObjectStore.Read(f"{self.ProjectId}/string_key")
JSON
To read a JSON object, call the ReadJson<T>
method.
var jsonData = ObjectStore.ReadJson<Dictionary<string, int>>($"{ProjectId}/jsonKey");
XML
To read an XML-formatted object, call the ReadXml<T>
method.
var xmlData = ObjectStore.ReadXml<XElement>($"{ProjectId}/xmlKey");
If you created the XML object from a dictionary, reconstruct the dictionary.
var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));
Delete Data
Delete objects in the Object Store to remove objects that you no longer need. If you run algorithms in QuantConnect Cloud, you need storage delete permissions to delete data from the Object Store.
To delete objects from the Object Store, call the Delete
method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.
if (ObjectStore.ContainsKey(key)) { ObjectStore.Delete(key); }
if self.ObjectStore.ContainsKey(key): self.ObjectStore.Delete(key)
To delete all of the content in the Object Store, iterate through all the stored data.
foreach (var kvp in ObjectStore) { ObjectStore.Delete(kvp.Key); }
for kvp in self.ObjectStore: self.ObjectStore.Delete(kvp.Key)
Cache Data
When you write to or read from the Object Store, the algorithm caches the data. The cache speeds up the algorithm execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:
- You open project A and save data under the key
123
. - You open project B and save new data under the same key
123
. - In project A, you read the Object Store data under the key
123
, expecting the data from project B, but you get the original data you saved in step #1 instead.
You get the data from step 1 instead of step 2 because the cache contains the data from step 1.
To clear the cache, call the Clear
method.
ObjectStore.Clear();
self.ObjectStore.Clear()
Storage Quotas
If you run algorithms locally, you can store as much data as your hardware will allow. If you run algorithms in QuantConnect Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.