QuantConnect Community

Good discussion here Guy Fleury and Peter Guenther. While also not saying that this strategy is good or bad, Guy Fleury makes an irrefutable point that what matters is the actual, or out-of-sample performance, and we can agree that 2022 was beyond awful for the strategy as it existed in Nov 2021. 

At least part of this is because, ironically, a longer backtest does not make a better backtest. And, CAGR over the entire period is actually a pretty poor measure of performance. It is much better to test a strategy (this goes for any model based on historical data, not just trading models) using training sets, validation sets and hold-out sets looking at many different such stratification selections, and then using a performance metric that incorporates both the return and drawdown in its calculation (or, for example, a metric that maximizes CAGR while minimizing the year-to-year CAGR variance). As it was, the 2008-2021 backtest ranked by CAGR couldn't help but overfit: much better would have been to look at several 3-year periods within that timeframe, tuning the model for its performance on other (not necessarily immediately subsequent) periods, looking for a set of results that achieved robust performance regardless of the validation time period.

In addition to the testing methodology described above, I keep coming back to what I see as a more fundamental question of algorithm design… the simply stated (though sometimes difficult to achieve) exercise of looking for holes or “what will kill the thing?” In this case, as I said in earlier comments, I believe that the twin assumptions of 1) continuing negative correlation between equity returns and bond returns; and 2) the continuing assumption of low inflation were the likely culprits. 

Now that we have a stress scenario (this year) with those to assumptions blown, it would now be useful to choose a timeframe, like Jan 2019 - Sept 2022 as our training set, and then go back and test the model on segments of the 2008-2020 timeframe. Might be that we get decent results, might be we don't; I haven't tried it yet. And then we can ask ‘what will now kill this thing?’ and see what we might come up with. My standard go-to in this situation is stagflation ala Japan of the last 30 years… if that happened, how might we expect the model to respond?

That sounds sensible, Mark Kust. Are you working on these kinds of tests by any chance at the moment, and would you be happy to share results in this thread? That would be very interesting material to look at.

As mentioned before, I propsed the following two changes:

• do not trade TMF or TLT if there is stat. significant yield inversion
• do not trade TMF or TLT if inflation is over 6%

it avoids most of the 2022 drawdown and seems not to impact performance 2008-2022. However, I forgot to do a performance analysis on 2002-2010. 

Regarding performance metric I am still struggling a lot to find a good solution. To compare performance of regimes/samples I currently use

• 95% confidence intervall of 1-y rolling sortino ratio (in daily steps)
• 95%  confidence intervall of 1-y rolling (compounding annual return % / max-drawdow.totalperiod)

Can you please share the algo with your changes Santa24 ?

I think that people have different argumentation angles here. A quick metaphor might help to disentangle. 
Let’s use a car: 
> Guy Fleury asserts that the car has crashed and concludes that the engine (the signals) is faulty and an airbag is required.
> I note that the engine was working fine and that the car’s features are the engine, a gas pedal (in selection), and a brake pedal (out selection). I argue and show that it was the brake pedal, not the engine, that was not working properly.
> Santa24 notes that the brake pedal can be fixed by a certain means.
> Mark Kust argues that the car needs to be tested more, e.g. on sub-sections of the road or completely different roads but maybe even on ice or water. He proposes that the car is only fully reliable if it can master all these different conditions.

I just need to drive the latest car  😊 

Yes, the engine failed, the car crashed (it was totaled) and there were no airbags. But airbags might not have been sufficient. Already, the composite signal of the In & Out strategy had for primary objective to protect from drawdowns. It was even the strategy's mantra: signal goes down, you get out of stocks; signal goes up, you move back in. For when the market was not moving up, you were in bonds or cash equivalents with low return expectations. Over last year, the signal (engine or whatever) was wrong 86% of the time. That makes the signal composition assumptions faulty and in need of repair, redesign, or at least reassessment.

I currently see the In & Out strategy as a classic case of curve fitting to the extreme. It had many modifications and tweaks while being developed on the Quantopian website. Its trading signals were fine-tuned over what should be considered its training and validation periods prior to October 2020. Based on portfolio metrics generated by those simulations prior to October 2020, it seemed to work fine, showed promise, including CAGR-wise.

This particular version was ported to QuantConnect in October 2020 and also had quite a few modifications. We could consider that from October 2020 to October 2021 (a whole year), the strategy was further refined and had its validation and walk-forward phase (a year is more than enough for that). Should we design these things to put them on walk-forward mode for years on end? At one point, those strategies have to go live or else be discarded.

In my previous post, I did not want to be too harsh which is why I used the term "walk forward" when describing the In & Out strategy. What I intended to use was "going live" after the strategy's one-year "walk forward" leading to October 2021. The strategy has been frozen in time since October 2021 and therefore any simulation up to October 2022 would be like if having traded this strategy live with its consequences (good or bad).

A simple simulation from October 2021 to October 2022 was sufficient to show that the strategy was clearly overfitted. It is the most reasonable explanation. From what should have been its live phase, the strategy went from an 86% win rate prior to October 2021 to a -86% loss rate from October 2021 to October 2022. This means that 86% of the trading signals were wrong or mistimed. Those were not sought-after portfolio metrics, especially if trading live.

Whatever this strategy was or has been up to October 2021, it does not matter so much for the person that traded that thing live from October 2021 to October 2022. It was simply a disaster, a real train wreck, a portfolio destroyer, providing no reprise.

What can be done now is a re-design, an overall restructuring. We could use the information that has been gained over those 14 years. But first, there is a need to demonstrate that the composite trading decision signal has value, not of the coincidental type once in a while but from a statistically significant point of view. However, there might not be sufficient data to make that case.

Here is the equation for this strategy, or any other, for that matter:

Guy Fleury the engine did not fail, unless this engine is severely different from the in / out thread, i believe the signals are the same. Algo versions I've posted in that thread only have a -11% drawdown for this whole insane year. Obviously not every signal is going to be a winner no idea where you are getting -80% drawdowns from.Nor that it matters, because what matters is money management and drawdowns, those algos in those threads are doing about 20% returns 20% drawdowns from 08 - now

@Jack, my comments have been about this particular version of the In & Out strategy.

I have not retested the other 4 versions to see their outcomes over the last year. They are basically different strategies that should give different results. Maybe some of their protective routines might help this version of the program. I do not know. But there, simulations and validation procedures will have to be done... to get an idea of what to expect going forward.

Your comment does raise a more interesting question: which strategies should one choose out of the 5 In & Out strategies or out of the tens of thousands out there? They will all give different results but you might be interested in only the few top performers going forward. We are still playing for the highest profit possible even with no guarantees.

If you find my presented data inaccurate, you can just redo the given simulation based on the October 2021 version which you will find here. You are expected, evidently, to get the same answers I got.

Here is also the URL for the results based on this version of In & Out over the last year only.

From its portfolio metrics, I cannot say it was a successful strategy with its -81% drawdown, its 86% loss rate, or its -1.27 Sharpe ratio. I do not think I would be alone to say this.

Definitely, something did not work out as planned... or even as expected...

Now, how can this strategy be repaired? Or is it even repairable?

The algorithm relied on bonds (TLT) which had an absolutely terrible run since YTD if you look at the charts.  Normally TLT is a safe haven during a recession, but not this time. The algorithm is still certainly still savlagable.  Instead of just SetHoldings to TLT, you could just do some balancing into a “defensive” bucket of stocks, similar to what a momentum algorithm does in the place of the self.HLD_OUT logic

I could see this getting combined with how this algorithm does a momentum strategy when picking which holding to take.  One could swap that out for  the Hold Out weight logic 
 

Glenn Vinson I am trying to run your algo live but keep getting into this error

Runtime Error: 'QLD' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("QLD") in Slice.cs:line 315 Stack Trace: 'QLD' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("QLD") in Slice.cs:line 315

Any clues?