Continuous Deep Reinforcement Learning on QC

Hi Joe, 

That is actually the next thing I wanted to implement here on QC. At the moment, I am going through a few books/references on DRL, but I should be coding up something in a few weeks. I will get back to this post when I have some results to discuss. In the meantime, thank you for sharing the code :)

Best wishes,

Lorenzo

Awesome! Thanks.

Hi Joe

Have you tried paper trading on QC hardware? The limitations I've found on this platform relate to the training time when deployed as it's limited. I've still not been able to get LEAN running well with python and DRL.

Interesting algo, well done on it's completion and for sharing so we can learn. 

Awesome share. Reinforcement learning seems interesting - any recommended books or resources on the theory?

On Ryan's point, I've hit some computational bottlenecks as well with deep learning nets on QC due to the time limit for training during backtests. Paper trading/live trading is fine since an hour/week is enough for training on a weekly or incremental basis, but during backtests an hour through 3-5 years of data just takes too long. Only way I could work around this was by splitting up backtests into small 6 month chunks, and doing some bookkeeping  via the ObjectStore by initializing states from the last chunk.

Ryan McMullan 

I am unsure how to save my Pytorch model to an ObjectStore which is needed if we want to save the trained model for live trading.  I did add the ability to save the replay buffer to an ObjectStore instance so in theory we can continue to train on new data.

Adam W 

Thanks, I agree there are a lot of computational bottlenecks.  We need a speedy GPU for training.  Have you figured out how to save a Pytorch model to the Objectstore? 

It's unlikely that any external model formats can be directly saved to the ObjectStore (maybe security reasons?), but perhaps you can save the relevant aspects of the model and serialize that into a compatible format. 

I'm not very familiar with reinforcement learning so can't comment much on the specifics here, but for instance a deep neural network can be characterized entirely by its architecture, layer weights, and internal states. To "save" my models to the ObjectStore, I extract the weights/states, serialize them into JSON, dump into ObjectStore, then "load" models by re-building the model with the pre-trained weights/states in the same architecture.  Perhaps a similar methodology could work here.

I'll see what we can do to make it possible.

All disregard if this is an unintelligent comment. I know a tiny bit on RL, first I've come into real contact with TD3 but very interesting after doing a bit of reading up. 

As more of a thought provoking suggestion and something to complicate it more, but possibly make it more advanced as far as your sediment tokenization. In the Tiingo News and Sediment.py section, what's to keep you from also incorporating a NLP Transformer like Bert or mid weight GPT-2? It could add more words  I've not seen it done with pytorch, but assume TF to pytorch would be possible. Would be able to word score with Tiingo sentiment engine? Just theories here, sadly not much help on the construction front. 

Brandon Schleter You can add a transformer and incorporate it into the network as an input. Or we can just wait for GPT-4 and just ask it what the best stocks to buy. lol

All kidding aside, I just took the sediment score and divided by a large number for an input. I wish I had more time to test things like you mentioned

 Joe Bastulli good one. I even feel like GPT-2, even mid tier level would still be computationally expensive layered into this, would likely run something seperate. GPT-4 will just read my Ikea manual, put together my furniture, and self complete algorithms. 

No worries, and while I'm a novice, I'll try a bit of things in relation to this, to try to use it for sentiment additions. 

Hello Joe Bastulli , i would like to know in which part of the project the permitted action space of the agent is declared and where the agent is doing the trade. Also if the action space is continous since TD3 policy gradient only works for continous action  space. Thanks.

Hey all - late to the thread here, but I wanted to share my solution for saving-loading models.

I train in the notebook, save the model, and then load the model when backtesting. I haven't gotten far enough to need automated training in the backtest code.

Here's a thread where I detail how to save/load using PyTorch:

And here is a thread where I ask Jared Broad about GPU training on QC (still not fully answered):

Overall, I'm torn - I want to use QC because of the data and backtesting capabilities, but I'm unable to train any heavy models due to 1. the lack of GPUs and 2. lack of visibility into my resource consumption - sometimes my models freeze during training and I suspect it's got something to do with memory consumption, but I can't tell for sure.

Best,