There are certain rules (like > 80% PSR) that are required for alpha market strategies for backtests.
As usual, many strategies are curve-fitted and no longer remain inside those parameters when traded live as can clearly seen from alpha market statistics (from hundreds of algos in alpha market, 80% PSR filter is only true for about 10 strategies)
So my question is two-fold:
- Is there an actual reason to use this tightly controlled parameters for backtests? Wouldn't filters like this encourage curve fitting especially as the backtest length is capped to only short period?
- If filters like these are realistic, shouldn't the same filters also apply to forward tests (ie. live statistics)? What is the point of only filtering strategies based on backtest?
My 2 centrs: I personally would much prefer longer backtests (20 years or so+) so I could see how the strategies work outside the current bull market and would like the filtering to be realistic so that I know that the filters are reasonable and that live results would also be filtered using same approach to discourage curve fitting.
Jared Broad
It is a tricky one. I agree in theory, but in practice, we were getting hundreds of variants of EMA cross strategies - very obviously overfit. By implementing the PSR filter we removed most of those strategies and helps see the “signal through the noise” with its statistical approach to reviewing the daily returns vs a human's ability to judge from a chart.
I agree it will be implemented in live trading as well, and I expect many will be cut. It just takes time as it's hard to analyze on the live charts.
I am currently thinking of a third way of approaching it - a “frozen backtest” only out-of-sample period for maybe 1-2 months which would be much cheaper to handle; and then deploying the ones which match their in-sample periods to paper trading. Then filtering the ones who stop performing as expected from paper as well.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Jared Broad,
I have other tricky questions in another to Mikko M's direction.
Why a 1-minute research strategy without any parameters has such excellent metrics?
Sharpe Ratio
2.167
Drawdown
0.200%
PSR
98.976%
Win Rate
100%
Beta
0
Annual Standard Deviation
0.005
Treynor Ratio
28.546
Estimated Strategy Capacity
$2900000.00
It is profitable over the course of the backtest.
It has Probabilistic Sharpe Ratio (PSR) of greater than 80%.
Correlation to other my strategies is less than 85%.
Is it qualified for alpha market?
How are you going to detect such "alphas"?
Maybe the word “excessive” is missing from the calculation of the Sharpe ratio?
Derek Melchin
Hi Vladimir,
The algorithm above does not qualify for the alpha market because it only fills 3 orders over a 5 year backtest. The algorithm buys SHV on May 12, 2016 and holds until May 5, 2021. We aim for an average of 10 trades per month. For a list of the alpha criteria, refer to the docs here.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Hi Derek,
What about this, actually the same, but with 14909 trades.
Sharpe Ratio
4.587
Total Trades
14909
Drawdown
0.200%
PSR
100.000%
Beta
-0.002
Annual Standard Deviation
0.002
Derek Melchin
Hi Vladimir,
The Alpha Team is reviewing the criteria for accepting alphas. The algorithm above uses hourly data but minute-resolution data will be required because it provides more accurate fill modeling with the bid/ask spread.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Derek,There were three reasons for publishing these two algorithms.
1. Draw QC's attention to the fact that excellent Sharpe ratio, total trades, drawdown, PSR, beta, annual standard deviation are useless if the information ratio is negative..
2. Maybe the word “excessive” is missing from the calculation of the Sharpe ratio?
3. I support Mikko M's request to lower PSR requirements if the information ratio is sufficiently positive.
Derek Melchin
Hi Vladimir,
Thank you for the feedback. We've forwarded this to the Alpha Streams review team.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Hi Derek,
How about my second question
2. Maybe the word “excessive” is missing from the calculation of the Sharpe ratio?
Derek Melchin
Hi Vladimir,
Please rephrase the question to clarify. The Sharpe ratio is currently calculated using a risk-free rate of 0. Refer to the source code here.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Jared Broad, Derek Melchin,
In definition of Sharpe Ratio (originally reward-to-variability ratio) named after the winner of the 1990 Nobel Memorial Prize William Forsyth Sharpe there is risk free return.
Now I understand why algorithm with Compounding Annual Return 0.940%,
(not making any excessive return to risk free rate) may have
Sharpe Ratio 4.122 and PSR 100.00%
Don’t You think that QC calculation is not The Sharpe Ratio at all?
The next question will be about calculation of the Information Ratio in
Alpha Market All Strategies page.
The information ratio and the Sharpe ratio are similar.
Both ratios determine the risk-adjusted returns of a security or portfolio.
However, the information ratio measures the risk-adjusted returns relative to a benchmark while the Sharpe ratio compares the risk-adjusted returns to the risk-free rate.
Today I sorted all strategies by score and found one on 8th place.
Waikikamukau v1.0 c5a00d5f82157bb94136c581f/v1.0
How come the algorithm with Compounding Annual Return 1.86%
can have positive Information Ratio of 0.32 ?
What benchmark QC uses to calculate Information Ratio?
BTW How QC computes the score?
Derek Melchin
Hi Vladimir,
> Don’t You think that QC calculation is not The Sharpe Ratio at all?
The standard proxy for the risk-free rate is the 90-day Treasury bill rate. The latest rate reported for the 13-week treasury bill was 0.03%. This is very close to 0 and changes over time. Using a standard 0% risk-free rate allows the Sharpe calculation to remaining consistent over time.
> How come the algorithm with Compounding Annual Return 1.86% can have positive Information Ratio of 0.32 ?
This is because the alpha outperformed the benchmark during the backtest period.
> What benchmark QC uses to calculate Information Ratio?
We use the benchmark that's set in the algorithm with the SetBenchmark method, which is SPY. Refer to the source code here.
> How QC computes the score?
Score = PSR * Sharpe * Capacity * ( MIN( daysLive/180, 1) )
Hi Mikko,
Thanks for the feedback. We've forwarded this to the Alpha Streams review team.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir
Derek Melchin
Thank you for detailed information.
->This is because the alpha outperformed the benchmark during the backtest period.
It should have been, but it didn't.
Vladimir
Derek Melchin,
The only 9 from 280 or 3.21% currently have PSR > 80.
Please forward this to the Alpha Streams review team.
Louis Szeto
Hi Vladimir
I’m no staff but I think it’s live statistics shown, not backtested ones, so alpha might got accepted if PSR > 80% in backtest. And its requirements were stricter from time to time, so some of that might pass the requirements in the past.
And the alpha you screenshot indeed better than benchmark in backtest period.
Cheers
Louis
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
Our implementation of the Probabilistic Sharpe Ratio is indeed for the backtest period that the probability of the out of sample being greater than 1.0 Sharpe. Many of the ones highlighted have live track records > 1.0 Sharpe, so the probability of the out-of-sample being > 1.0 is 100%.
I appreciate the suggestions and we will get to filtering in live trading, but it's a balance because if we're too aggressive we'll have no alphas in the market.
We also keep many of them for prolonged periods after obvious downturns for transparency and to show the regime change. Many worked in live trading out-of-sample for a time and then stopped. Not all alphas will work all the time, we need to have a suite of 1,000 alphas for various regimes which get spun up when appropriate.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mikko M
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!