Cloud Platform

Data Storage

Introduction

The ObjectStore is an organization-specific key-value storage location to save and retrieve data in QuantConnect's cache. Similar to a dictionary or hash table, a key-value store is a storage system that saves and retrieves objects by using keys. A key is a unique string that is associated with a single record in the key-value store and a value is an object being stored. Some common use cases of the ObjectStore include the following:

  • Transporting data between the backtesting environment and the research environment.
  • Training machine learning models in the research environment before deploying them to live trading.

The ObjectStore is shared across the entire organization. Using the same key, you can access data across all projects in an organization.

Supported Types

The ObjectStore has helper methods to store strings, JSON objects, XML objects, and bytes.

The ObjectStore has helper methods to store strings and bytes.

ObjectStore.Save(stringKey, stringValue);
ObjectStore.SaveJson<T>(jsonKey, jsonValue);
ObjectStore.SaveXml<T>(xmlKey, xmlValue);
ObjectStore.SaveBytes(bytesKey, bytesValue);
self.ObjectStore.Save(string_key, string_value)
self.ObjectStore.SaveBytes(bytes_key, bytes_value)

To store an object that is in a different format, you need to encode it to one of the supported data types. For instance, if you train a machine learning model and it is in binary format, encode it into base 64 before saving it.

The ObjectStore also has helper methods to retrieve the stored objects.

var stringValue = ObjectStore.Read(stringKey);
var jsonValue = ObjectStore.SaveJson<T>(jsonKey);
var xmlValue = ObjectStore.SaveXml<T>(xmlKey);
var bytesValue = ObjectStore.SaveBytes(bytesKey);
string_value = self.ObjectStore.Read(string_key)
bytes_value = self.ObjectStore.ReadBytes(bytes_key)

For complete examples of using the ObjectStore, see Object Store.

Storage Sizes

All organizations get 50 MB of free storage in the ObjectStore. Paid organizations can subscribe to more storage space. The following table shows the cost of the supported storage sizes:

Storage Size (GB)Monthly Cost ($)
0.050
210
520
1050
50100

Research to Live Considerations

When you deploy a live algorithm, you can access the data within minutes of modifying the ObjectStore. Ensure your algorithm is able to handle a changing dataset.

The live environment's access to the ObjectStore is much slower than in research and backtesting. Limit the individual objects to less than 50 MB to prevent live trading access issues.

Monitor Usage

The Resources page shows the total storage used in your organization and the storage used by individual projects so that you can easily manage your storage space. To view the page, log in to the Algorithm Lab and then, in the left navigation bar, click Organization > Resources.

Organization Object Store usage panel

Edit Storage Plan

You need storage billing permissions and a paid organization to edit the size of the organization's ObjectStore.

Follow these steps to edit the amount of storage available in your organization's ObjectStore:

  1. Log in to the Algorithm Lab.
  2. In the left navigation bar, click Organization > Resources.
  3. On the Resources page, scroll down to the Storage Resources and then click Add Object Store Capacity.
  4. On the Pricing page, select a storage plan.
  5. Click Proceed to Checkout.
  6. Project Object Store limit panel

Delete Storage

To free up storage space, delete the key-value pairs in the ObjectStore by calling the Delete method with a key.

ObjectStore.Delete(key);
self.ObjectStore.Delete(key)

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: