Offering a faster alternative to calling Self.History(). In my tests, I've seen up to 10x speed gains; you may or may not see the same, depending on your use case. Eg: expect different different performance gains if you are fetching a single history bar for one asset, versus thousands of bars for hundreds of assets.
I'm calling it an Identity History indicator. It's not really an "indicator" per se, but leverages the indicator architecture, extending from the PythonIndicator base class. It has “Identity” functionality in that it doesn't perform any calculations – it just gives you back the price information, just like the QC Identity indicator. The difference is it gives you historical information, not just the most recent.
Hope this helps someone.
How to Use it:
Create an instance of the indicator, then warm it up using self.WarmUpIndicator() with your desired resolution. Once it's ready you can access the 'History' property of the indicator, which contains a list of the history bars. An example is in the attached backtest.
Why it works:
Calling Self.WarmupIndicator() is faster than Self History() because the history request and the indicator update are done on the "C# side" instead of on the Python side, and the history requests don't need to be converted to pandas and then back to TradeBar.
Notes:
- You can only fetch historical data from some point in history up till present. With Self.History you can get fetch historical data between to arbitrary dates.
- There are even further performance improvements to be made, like using a list instead of a dequeue (thanks Fred Painchaud ) – I may get around to sharing an updated version soon.
Thanks to Fred Painchaud and Alexandre Catarino for validating / bouncing ideas around this effort.
Arthur Asenheimer
Thanks for sharing.
Interesting idea to use custom indicators and WarmUpIndicator to fetch historical data. :-)
I doubt a list will be faster here as deque is the optimized data structure for doing append/pop on both ends and you're doing this many times in the Update method of your custom indicator.
Did you notice you can also fetch historical data directly as a list using self.History[TradeBar](…) ? It's not 10x faster but still 3-4x faster.
Fun fact: Using History[TradeBar]() and then transforming it to a pandas DataFrame via list comprehension is still 2x faster compared to calling History() directly which indicates that PandasConverter.GetDataFrame() is not very efficient in Python. See also research notebook attached.
Arthur Asenheimer
By the way, I ran a few backtests and I don't see any performance gains when using your FetchHistoryWithIndicator version. It actually seems to be slower. :-/
.ekz.
Thanks for the tip, Arthur Asenheimer! Very interesting findings. Alexandre Catarino did also mention the TradeBar casting would help, but I found the warmup faster still.
That's odd that you are seeing it perform slower :(
Can you please share more about your particular test case?
Arthur Asenheimer
There is nothing special about my “particular” test case. I just cloned your algorithm and ran a backtest.
Gave it another try and still don't see any performance gains. However, the duration varies a lot.
No code changes were made. Attached the associated backtest for the sake of transparency.
.ekz.
Wow… you are correct. Cloning this I saw the same.
Not sure if something changed since I did the tests (some time ago). I wonder if it's because it's in isolation, as it's faster in my actual system.
On the road now but will revisit this.
Stay tuned.
.ekz.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!