Back

Analyzing the backtests results issue

Hey Folks:

These days some posts were about the problem of analyzing the backtests results, and the need of more log.

After thinking about it I realized that the engine already gives us all the information we need to fully analyze the backtest in the downloadable trades log. But that data need to be processed in order to obtain valuable information.

What can be considered valuable information? To begin, just the basics: the daily algorithm equity curve, the daily volume traded by stock, the daily result by stock. Maybe you can help me naming some others?

I see two options to extract the information from the trades data:
- Make a program to process the trades data.
- Make the engine generate the info and make it available just like the trades data: I think this is a better option because the engine already estimates those values through the rolling statistics. Or, at least, the info is easily available from Transaction and Trade QCAlgorithm properties.

So, what do you think?

I’m available to help in the development of this issue.
Update Backtest






The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



Awesome @Juan, I definitely agree all the data is there and its just a matter of processing it into nice charts and picking out which charts we'd like :)

Even better than the downloadable trade log is the full BacktestResultPacket -- this has the equity charts, daily profit loss and all the statistics you'll need. We want to make it so people can use this JSON object and load up all the information for a backtest.

I'd recommend we write extensions / enhancements to the Statistics class. This way the results will also be available on the website virtually immediately. Alternatively if your work requires special charting libraries you could add it to the toolbox project.

I like the colored daily/weekly distribution matrix idea, larger rolling sharpe, rolling drawdown charts (we don't have this calculated), top drawdown periods, distribution of returns. We need some more infrastructure to do "out of sample" charting.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey @Jared,

I really want to help but the topic is way above my head. I don’t even know what JSON is and certainly I can’t do any of the task listed here. I tried to figure out how the things works in debugging mode, and I quickly get lost in so many classes and calls.

Thinking out loud: I think I can make the calls to estimate the daily volume traded per stock but I’m not sure how that calls can fit in the engine.

Anyway, the equity data should be available to download as csv just like the trades logs; even if isn’t the best option.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey guys, you can now download the JSON object which generated your backtest result :) Hopefully this empowers you to make beautiful dashboards with even deeper analysis. You can do this by clicking on the "Download Result" button in the Share tab.

This object can be deserialized into a C# type 'BacktestResultPacket' using Newtonsoft.Json.

var resultJsonString = File.ReadAllText("my-backtest-result.json");
var results = JsonConvert.DeserializeObject(resultJsonString);


2

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


WOW! Just great. Thank you very much Jared and all the QC staff!

I was working in the calls I mentioned above, so far this is what I have:
var dailyStatisticsByStock = TradeBuilder.ClosedTrades
.GroupBy(trades => trades.Symbol,
(symbol, trades) => new
{
ticket = symbol,
dailyValues = trades
.GroupBy(trade => trade.ExitTime.Date,
(day, trade) => new
{
index = day.Date,
profit = trade.Sum(t => t.ProfitLoss),
volShares = trade.Sum(t => t.Quantity),
volDollars = trade.Sum(t => t.Quantity * (t.ExitPrice + t.EntryPrice))
// Any other daily statistic by stock.
}
)
});

Is pretty ugly, and I'm not sure if the volume is correctly estimated, but is a first working draft. Some of you will note the Pandas Panel way of thinking, i.e. there is a DataFrame for each symbol with the daily statistics.

As you can see only needs the TradeBuilder, so as far I understand, it can be called from anywhere in the engine.

I'll research how to correctly make the volume estimations the and how to serialize this object to JSON.

Cheers, JJ
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Percentage of time in cash could be interesting, perhaps this can be calculated from the trades.

We have a daily performance gain/loss chart under the name "Daily Performance". This could be used for summing monthly PnL.
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Hey @Jared,

In order to estimate the percentage of time in cash, I'm thinking in use the Time Period library (available in NuGet); more specifically, the TimeGapCalculator . The idea is defining a time period for each trade in the TradeBuilder.ClosedTrades using the EntryTime and ExitTime and estimates the gap. Once the gap is estimated, is jut matter of make a division to obtain the percentage of time in cash. What do you think?

Edit: I must incorporate all the open trades too!
0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed