Research Environment

Object Store

Introduction

The Object Store is a file system that you can use in your algorithms to save, read, and delete data. The Object Store is organization-specific, so you can save or read data from the same Object Store in all of your organization's projects. The Object Store works like a key-value storage system where you can store regular strings, JSON encoded strings, XML encoded strings, and bytes. You can access the data you store in the Object Store from backtests, the Research Environment, and live algorithms.

When you deploy live algorithms, the state of the Object Store is copied, but it never refreshes. Therefore, if you save data in the Object Store in a live algorithm, you can access the data from the live algorithm, backtests, and the Research Environment. However, if you save content into the Object Store from the Research Environment or a backtest after you deploy a live algorithm, you can't access the new content from the live algorithm.

Get All Stored Data

To get all of the keys and values in the Object Store, iterate through the ObjectStore object.

foreach (var kvp in qb.ObjectStore)
{
    var key = kvp.Key;
    var value = kvp.Value;
}
for kvp in qb.ObjectStore:
    key = kvp.Key
    value = kvp.Value

To get iterate through just the keys in the Object Store, iterate through the Keys property.

foreach (var key in qb.ObjectStore.Keys)
{
    continue;
}
for key in qb.ObjectStore.Keys:
    continue

Create Sample Data

You need some data to store data in the Object Store.

Follow these steps to create some sample data:

  1. Create a dictionary.
  2. var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };
  3. Create a string.
  4. var stringSample = "My string";
    string_sample = "My string"
  5. Create a Bytes object.
  6. var bytesSample = Encoding.UTF8.GetBytes("My String");
    bytes_sample = str.encode("My String")
  7. Convert the dictionary to an XML-formatted object.
  8. var xmlSample = new XElement("sample",
        dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value)));
    Console.WriteLine(xmlSample.ToString());

Get File Path

To get the file path for a specific key in the Object Store, call the GetFilePath method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.

var filePath = qb.ObjectStore.GetFilePath(key);
file_path = qb.ObjectStore.GetFilePath(key)

Save Data

The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment.

If you run algorithms in QC Cloud, you need storage create permissions to save data in the Object Store.

If you don't have data to store, create some sample data.

You can save the following types of objects in the Object Store:

  • Bytes objects
  • string objects
  • JSON objects
  • XML-formatted objects

To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

You can save Bytes and string objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

Bytes

To save a Bytes object, call the SaveBytes method.

var saveSuccessful = qb.ObjectStore.SaveBytes($"{qb.ProjectId}/bytesKey", bytesSample)
save_successful = qb.ObjectStore.SaveBytes(f"{qb.ProjectId}/bytes_key", bytes_sample)

Strings

To save a string object, call the Save or SaveString method.

var saveSuccessful = qb.ObjectStore.Save($"{qb.ProjectId}/stringKey", stringSample);
save_successful = qb.ObjectStore.Save(f"{qb.ProjectId}/string_key", string_sample)

JSON

To save a JSON object, call the SaveJson<T> method. This method helps to serialize the data into JSON format.

var saveSuccessful = qb.ObjectStore.SaveJson<Dictionary<string, int>>($"{qb.ProjectId}/jsonKey", dictSample);

XML

To save an XML-formatted object, call the SaveXml<T> method.

var saveSuccessful = qb.ObjectStore.SaveXml<XElement>($"{qb.ProjectId}/xmlKey", xmlSample);

Read Data

To read data from the Object Store, you need to provide the key you used to store the object.

You can load the following types of objects from the Object Store:

  • Bytes objects
  • string objects
  • JSON objects
  • XML-formatted objects

You can load Bytes and string objects from the Object Store.

Before you read data from the Object Store, check if the key exists.

if (qb.ObjectStore.ContainsKey(key))
{
    // Read data
}
if qb.ObjectStore.ContainsKey(key):
    # Read data

Bytes

To read a Bytes object, call the ReadBytes method.

var bytesData = qb.ObjectStore.ReadBytes($"{qb.ProjectId}/bytesKey");
byte_data = qb.ObjectStore.ReadBytes(f"{qb.ProjectId}/bytes_key")

Strings

To read a string object, call the Read or ReadString method.

var stringData = qb.ObjectStore.Read($"{qb.ProjectId}/stringKey");
string_data = qb.ObjectStore.Read(f"{qb.ProjectId}/string_key")

JSON

To read a JSON object, call the ReadJson<T> method.

var jsonData = qb.ObjectStore.ReadJson<Dictionary<string, int>>($"{qb.ProjectId}/jsonKey");

XML

To read an XML-formatted object, call the ReadXml<T> method.

var xmlData = qb.ObjectStore.ReadXml<XElement>($"{qb.ProjectId}/xmlKey");

If you created the XML object from a dictionary, reconstruct the dictionary.

var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));

Delete Data

Delete objects in the Object Store to remove objects that you no longer need. If you use the Research Environment in QC Cloud, you need storage delete permissions to delete data from the Object Store.

To delete objects from the Object Store, call the Delete method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.

if (qb.ObjectStore.ContainsKey(key))
{
    qb.ObjectStore.Delete(key);
}
if qb.ObjectStore.ContainsKey(key):
    qb.ObjectStore.Delete(key)

To delete all of the content in the Object Store, iterate through all the stored data.

foreach (var kvp in qb.ObjectStore)
{
    qb.ObjectStore.Delete(kvp.Key);
}
for kvp in qb.ObjectStore:
    qb.ObjectStore.Delete(kvp.Key)

Cache Data

When you write to or read from the Object Store, the notebook caches the data. The cache speeds up the notebook execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:

  1. You open project A and save data under the key 123.
  2. You open project B and save new data under the same key 123.
  3. In project A, you read the Object Store data under the key 123, expecting the data from project B, but you get the original data you saved in step #1 instead.
  4. You get the data from step 1 instead of step 2 because the cache contains the data from step 1.

To clear the cache, call the Clear method.

qb.ObjectStore.Clear();
qb.ObjectStore.Clear()

Storage Quotas

If you use the Research Environment locally, you can store as much data as your hardware will allow. If you use the Research Environment in QC Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: