Discussion Forum - QuantConnect.com

Gregory Szczeszynski INVESTOR

| Reward Discussion

I thought I'd post how to save DataFrames to the ObjectStore and restore them since it took me a while to get it working (not obvious).

The only way that the round-trip seems to work with is to have the orientation set to "records":

# Save DataFrame data into object store

json_string = stock_history.to_json(orient='records')

qb.ObjectStore.Save("Last_Stock_History", json_string)



# Read back data from object store

json_string = qb.ObjectStore.ReadString("Last_Stock_History")

stock_history_new = pd.read_json(json_string, orient='records')

The main problem here is that the indexes are lost. You will need to reset_indexes and then recover them after:

# Flatten DataFrame (move all indexes into column data)

# Need to do this before converting to json_string

stock_history = my_dataframe.reset_index()





# Once you have loaded the data from the store you can restore the indexes

# Need to know what the original indexes were though

my_dataframe = stock_history.set_index(['symbol','time'])

Update Backtest

Gregory Szczeszynski

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Gregory Szczeszynski

160 ,

One update: the date-time column is not converted properly, so the following will clear that up:

# Save DataFrame data into object store 

tmp  = stock_history.reset_index()   # Flatten indexes into data columns

json_string = tmp.to_json(orient='records', date_unit='ns') 

qb.ObjectStore.Save("Last_Stock_History", json_string)





# Read back data from object store 

json_string = qb.ObjectStore.ReadString("Last_Stock_History")

stock_history_new = pd.read_json(json_string, orient='records')



# Restore the "date" columns - in this example only 'time' is a date

stock_history_new['time'] = pd.to_datetime(stock_history_new['time'])



# Restore indexes

stock_history_new.set_index(['symbol', 'time'], inplace=True)

Derek Melchin

STAFF ,

Hi Gregory,

Thanks for sharing. We also need to recreate the Symbol in the DataFrame index. For completeness:

# Save to ObjectStore

qb.ObjectStore.Save("data", history.reset_index().to_json(date_unit='ns'))



# Read back data from object store

restored_history = pd.read_json(qb.ObjectStore.Read("data"))



# Restore the indices

restored_history['time'] = pd.to_datetime(restored_history['time'])

restored_history['symbol'] = restored_history['symbol'].apply(lambda x: qb.Symbol(x))

restored_history.set_index(['symbol', 'time'], inplace=True)

See the attached notebook for reference.

Best,
Derek Melchin

Gregory Szczeszynski INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240600 Quants.

Online Now

Stay Connected

How to save DataFrames in Research to ObjectStore

Started By:

Join QuantConnect for Free Sign Up

Platform

SIGN IN

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240600 Quants.

Online Now

Stay Connected

How to save DataFrames in Research to ObjectStore

Started By:

Join QuantConnect for Free Sign Up