Analyzing the backtests results issue

| |

Hey Folks:

These days some posts were about the problem of analyzing the backtests results, and the need of more log.

After thinking about it I realized that the engine already gives us all the information we need to fully analyze the backtest in the downloadable trades log. But that data need to be processed in order to obtain valuable information.

What can be considered valuable information? To begin, just the basics: the daily algorithm equity curve, the daily volume traded by stock, the daily result by stock. Maybe you can help me naming some others?

I see two options to extract the information from the trades data:

- Make a program to process the trades data.

- Make the engine generate the info and make it available just like the trades data: I think this is a better option because the engine already estimates those values through the rolling statistics. Or, at least, the info is easily available from Transaction and Trade QCAlgorithm properties.

So, what do you think?

I’m available to help in the development of this issue.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Awesome @Juan, I definitely agree all the data is there and its just a matter of processing it into nice charts and picking out which charts we'd like :) Even better than the downloadable trade log is the full BacktestResultPacket -- this has the equity charts, daily profit loss and all the statistics you'll need. We want to make it so people can use this JSON object and load up all the information for a backtest. I'd recommend we write extensions / enhancements to the Statistics class. This way the results will also be available on the website virtually immediately. Alternatively if your work requires special charting libraries you could add it to the toolbox project. I like the colored daily/weekly distribution matrix idea, larger rolling sharpe, rolling drawdown charts (we don't have this calculated), top drawdown periods, distribution of returns. We need some more infrastructure to do "out of sample" charting.

Jared Broad

STAFF Pro ,

JayJayD

42.9k Pro ,

Hey @Jared, I really want to help but the topic is way above my head. I don’t even know what JSON is and certainly I can’t do any of the task listed here. I tried to figure out how the things works in debugging mode, and I quickly get lost in so many classes and calls. Thinking out loud: I think I can make the calls to estimate the daily volume traded per stock but I’m not sure how that calls can fit in the engine. Anyway, the equity data should be available to download as csv just like the trades logs; even if isn’t the best option.

Hey guys, you can now download the JSON object which generated your backtest result :) Hopefully this empowers you to make beautiful dashboards with even deeper analysis. You can do this by clicking on the "Download Result" button in the Share tab. This object can be deserialized into a C# type 'BacktestResultPacket' using Newtonsoft.Json.

var resultJsonString = File.ReadAllText("my-backtest-result.json");
var results = JsonConvert.DeserializeObject(resultJsonString);

WOW! Just great. Thank you very much Jared and all the QC staff! I was working in the calls I mentioned above, so far this is what I have:

var dailyStatisticsByStock = TradeBuilder.ClosedTrades
                           .GroupBy(trades => trades.Symbol,
                                    (symbol, trades) => new
                                    {
                                        ticket = symbol,
                                        dailyValues = trades
                                        .GroupBy(trade => trade.ExitTime.Date,
                                                    (day, trade) => new
                                                    {
                                                        index = day.Date,
                                                        profit = trade.Sum(t => t.ProfitLoss),
                                                        volShares = trade.Sum(t => t.Quantity),
                                                        volDollars = trade.Sum(t => t.Quantity * (t.ExitPrice + t.EntryPrice))
                                                        // Any other daily statistic by stock.
                                                    }
                                                    )
                                    });

Is pretty ugly, and I'm not sure if the volume is correctly estimated, but is a first working draft. Some of you will note the Pandas Panel way of thinking, i.e. there is a DataFrame for each symbol with the daily statistics. As you can see only needs the TradeBuilder, so as far I understand, it can be called from anywhere in the engine. I'll research how to correctly make the volume estimations the and how to serialize this object to JSON. Cheers, JJ

Percentage of time in cash could be interesting, perhaps this can be calculated from the trades. We have a daily performance gain/loss chart under the name "Daily Performance". This could be used for summing monthly PnL.

Hey @Jared, In order to estimate the percentage of time in cash, I'm thinking in use the Time Period library (available in NuGet); more specifically, the TimeGapCalculator . The idea is defining a time period for each trade in the TradeBuilder.ClosedTrades using the EntryTime and ExitTime and estimates the gap. Once the gap is estimated, is jut matter of make a division to obtain the percentage of time in cash. What do you think? Edit: I must incorporate all the open trades too!

Bruce Vanstone

231 Pro ,

Can you please give a python example of how to do this... and what the name of the backtest object is after I have run a backtest.

I am quite happy to contibute time to developing advanced analytics, but it is unclear how to obtain the data to start with,

cheers,

Bruce

Derek Melchin

Hi Bruce,

To get the backtest results into the research environment, we can use

backtest = api.ReadBacktest(projectId, backtestId)

We are working on tools to help process the `backtest` result object. For now, we can access the Sharpe ratio for example with

backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio

See the attached research notebook for an example.

The backtestId can be seen under the "Share" tab of the backtest results page

Best,
Derek Melchin

Tzuo Hann Law

143 Pro ,

Hi I just tried to replicate this, but it did not work. Can someone confirm that this is functional please?

Arthur Asenheimer

17.6k Pro ,

It didn't work for me, too.

The commands api.Connected and backtest = api.ReadBacktest(projectId, backtestId) were successful, but backtest.Result yields a NoneType object.

Erol Aspromatis

99 Pro ,

I'm getting the same error, the API is not finding my backtest and retuirning it, rather it is creating a empty object.

AttributeError Traceback (most recent call last) <ipython-input-15-b8bc2d3efee3> in <module> ----> 1 backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio AttributeError: 'NoneType' object has no attribute 'TotalPerformance'

Sorry about we made some fast-breaking changes before the holiday weekend and will push the fixes for them today. We move quickly and there will always be bugs in complex software. Microsoft is still patching Windows since 1989 =). But please respect the forum etiquette and not post bug reports to the forum -- please submit all bug reports to support@quantconnect.com so we can keep the forum focused on algorithm development.

JayJayD INVESTOR

Update Backtest

Notebook

person upvoted this people upvoted this

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Alpha League Competition: $1,000 Weekly Prize Pool

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research PUblications

261,400 Quants.

VOTE FOR UPCOMING FEATURES

Analyzing the backtests results issue

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Alpha League Competition: $1,000 Weekly Prize Pool

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

TOP 5 Research PUblications

261,400 Quants.

VOTE FOR UPCOMING FEATURES

Analyzing the backtests results issue

IN THIS RESEARCH

PARTICIPANTS

Discussion Awards

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

Actions

Join QuantConnect for Free