QuantConnect Community

I know there's countless things that could be added into the fitness function and it's important to limit it to what's crucial and I think you've done a great job of that. I think that there should be a variable of time backtested added. Knowing a history has a significant amount of time that it has been tested on can help bolster some confidence that it will continue in the future. Just my two cents!

Thank you SLPAdvisory! We thought the same, the way we concluded to add time factor using "Portfolio Turnover Ratio x Years of Backtest" would give a ballpark indicator the period was sufficient. We're pushing this today in the backtesting view. e.g.

What is the fitness ranking cutoff for alpha streams approval? 

How is the high turnover ratio determined?

My fitness score and alpha rank are not updating for a certain algorithm.  Any idea what is causing them to be zero?  It was updating on a different algorithm I was using.

Hi Kieran Flynn , it is a "pass" if the average daily turnover of your strategy is greater than 1. This would mean with a $10,000 portfolio you'd need to do at least $10,000 in volume per day. 

I am concerned this might be too hard for more people to hit, and perhaps a more realistic target should be at least 30% portfolio turnover. Currently, the score is linearly scaled down to 0 based on this percentage. So if you only do 5% turnover per day it would scale score to 5.

Jon D. Mitchell just asked a question here which I think might be because of the strategy volume.

The key goal is 100% confidence the returns of the strategy are due to your alpha.

Jared,

Thanks for sharing these ideas with the community and requesting feedback.

Sortino and RoMaD are very good risk metrics.  The volume factor is concerning because it could easily be manipulated and doesn't necessarily provide a fair evaluation of the trading strategy.  

100% daily turnover, or any arbitrary daily turnover rate, favors intraday strategies that may not have desirable risk adjusted returns.  See the following 2 illustrative examples. In my opinion, most people would prefer the risk profile of the biweekly strategy even though it's score is lower than the intraday one.

1. Intraday:  sortino of .35; RoMaD of .25, volume of 1; Score = [1 * (.55+.39) + 10]/20 = .55

2. Biweekly:  assume portfolio only turns over once every 2 weeks (or once every 10 trading days).  Sortino = 2; RoMaD = 1.5; volume = .1; Score = [.1 * (2.67 + 2.14) + 10] / 20 = .52

Along the same lines as SLPAdvisory's suggestion, instead of volume, maybe a better approach would be to look at the risk metrics (Sortino and RoMaD) over a longer period of time, say 5 years, and penalize any strategy that does not provide at least that many years of backtest and/or live data.  For example, fitness = MIN(1, Trading Days/5 years) * Sortino over MIN(Trading Days, 5 years)

It also seems like the range of results is narrow. Most people would be very happy with a strategy that has a Sortino of 1.5 and RoMaD of 1 over a 5 year period; but the score for that would be .68. As a comparison, from 1/1/2014 to 12/31/2018, SPY had a Sortino of .83 and a RoMaD of .45 (assuming the risk free rate = 0), this would result in a score of .6; not too far off from the strategy with much better risk metrics.

My suggestion would be to provide each risk metric as individual filters (one for Sortino and one for RoMaD) on the Alpha stream page. Another metric that may be helpful is total return, or excess return over a risk free rate, to filter out strategies that have good risk measures but low total returns.

There will never be a perfect solution, but our clients are likely sophisticated and value having the ability to filter strategies based on their preferred metrics.  Have you considered asking some of the hedge or fund managers their view?

Thanks again.

I agree with the above comment a lot, I'm not a huge fan of the volume factor. I see it's merits which is why I really like the idea of having the score in all categories shown on the alpha page, but at least for our practice we utilize pretty infrequent trading strategies for the tax benefits as well as limiting trading costs. I have a feeling at least on the RIA advisor side they would be significantly more interested in less active strategies.

Thanks for the comments guys! And an interesting point about manipulating it by simply re-buying SPY each day. 

Just to be clear: funds have the insights, statistics and equity curve and its all delivered by API. The charts on the market page are for the community. The Alpha Streams goal is to allow automated scanning of the marketplace historical results and license in bulk. All of the algorithm's live signals are streamed by API as well. 

The fitness function is for QuantConnect. We are getting a lot of algorithms and what makes it into the market vs what gets bounced is too gray at the moment. I'd like to create a programmatic cut-off so the community can know what the market place is seeking. Imagine we can only host 1,000 alpha stream strategies -- which 1,000 should make the cut? How do you distinguish without imposing bias?

Ultimately the goal of the volume component was to programmatically determine the strategy is trading in a statistically significant manner so we can attribute the performance to the owner. For HFT strategies "statistical significance" can be seen in a 3 month backtest so we don't want to impose a period like 5 years into the equation.

Do you have other ideas for filtering out the "Buy APPL in 1998" strategies? 

Re: SPY vs Intraday Tristan -- one community member suggested in private feedback to multiply the values of RoMaD and Sortino, and scale to the new distribution. This way the poor RoMaD of SPY would eliminate it as a high drawdown strategy. 

Also important to mention -- the fitness score running in production is slightly different to the one in the research notebook above - the key difference being Martin shifted the scaled values by 5 - this allowed algorithms which have an absolute value of 0 (e.g. doing no trading, or losing money) to start from 0 instead of 50%! 

The test I usually perform to account for overfitting or what my cover the "Buy AAPL or AMZN in 1998" is to just change the ticker. Especially if it's a intraday algorithm that is analyzing patterns or trends, I'm of the camp that it should be able to be applied to similar success across tickers. You could try substituting the ticker in the provided algorithm with a basket of indexes? Really glad to see your involvement with the community on this!

Jared Broad: Do you have other ideas for filtering out the "Buy APPL in 1998" strategies? 

I have got an idea. It is aimed more at the composition of the portfolio than the turnover. 

In my eyes a portfolio with a lot of different open positions is much more resilient than a portfolio with AAPL stocks only, especially when the open positions are little or even negative correlated to each other. 

where N is the number of open positions and c_ij are the correlations coefficients between each open position. So C is nothing else than the sum of all elements of the correlation matrix except the diagonal. To divide the sum by N^2 makes sure C is between -1 and 1 so it should be easy to shift the fitness function. 

The smaller C, the better the portfolio. 

By subtracting C from your current fitness function you punish "Buy AAPL in 1998" strategies with a worse score. 

On the other hand, a portfolio with a lot of negative correlated positions would get a better score. That's a good thing in my eyes because a portfolio with more negative correlated positions is less vulnerable in market turmoils.

Re: I've to admit that correlations can change with time but usually they persist for a long time. However, we should use something like the rolling 90 day correlation instead of using the whole price history. 

"Buy AAPL in 1998" yields 0 Fitness Score currently because there's zero turnover. I think these kinds of strategies will naturally ruin their own scores - it's hard to "cheat" meaningful performance with the kind of turnover needed to get a high score.

It's anyway highly likely that it will be necessary to manually reviewing every strategy meeting the initial score cut-off. "Cheaters" can be caught with relative ease this way I suspect.

Having said that, I don't think the current implementation of GetScaledPortfolioTurnover() is sound - you're rewarding strategies that turn over themselves entirely every day, or close to that, bringing about enormous trading costs and throwing any kind of scalability out the window. I'm assuming you have had advice from funds that this is what they look for, but frankly I'm really surprised this is implemented in this way.

Large quant institutions who will be the primary licensors have a bias towards active strategies as they can be quickly switched off when performance out of sample changes. However, the turnover was intended to be a minimum turnover for acceptance, I was concerned 1.0 might be too hard for most people to hit but a lower minimum of 0.3 might be more suitable. The linear function 0-1 @ MinTurnover is not designed to incentivize/reward any strategy but set that minimum volume, if the volume is a good signal for significance.

Thank you for the suggestion Arthur. We can't assume it is a portfolio based strategy - if you have an alpha which predicts orange juice futures alone its still a great factor for a multifactor portfolio. However, building off your idea perhaps we could use the correlation of the open positions to the equity curve. This would be a direct way of measuring the performance is due to the assets. 

Tristan F  -- SPY with Portfolio turnover of 2.0 -- score of 55.

Jared,

You can create a volume factor of 100% by being out of the market for 1 minute instead of over night.  See attached for score of 60.

Is the Sortino ratio available anywhere on backtest results?  I can't find it.

Thanks.

Right! And that score is actually 200% volume factor as each side of the trade contributes. This is why we set it to 1.0 to start with - its an easily achievable goal if you're actually trading. 

Thanks for clarifying Jared.  If you believe your clients will be focused on HFT trading type algorithms than maybe the volume factor works.  I still believe it will leave out some very good algorithms and that it's too easy to manipulate to be very useful.

Another point is that Sortino and RoMaD are measuring very similar things.  Sortino is return over negative deviation, and RoMaD is return over maximum drawdown, One measures the expected and the other the extreme downside risk.  What's the reason for including both?  Are there well known use cases where Sortino looks great but the RoMaD is terrible, or vice versa?

I think Sortino's negative deviation results in it being an average negative variance of the population (sum/count) which lets it hide martingale type strategies with 99% perfect trades and one 50% drawdown. The reason for including both was so that RoMaD would eliminate the absolute worst value drawdown "martingale-option-put selling" hidden risk algorithms.

Hi,

I'd add a criteria to this Fitness Function which would take into account the long running well performing backtests (see example below). Does it make sense? What do you think?

Here are other thoughts to gauge the value of trading and screen out "Buy APPL in 1998" strategies. This will take a few posts, my apologies in advance.

The two primary ways that can make active trading more profitable than buy-and-hold are:

1. Choice of assets

2. Trading pattern (buy and sell decisions)

Choice of assets

Here you would want to compare the assets you selected to an appropriate benchmark. The traditional CAPM model could provide an alpha that shows the value of the chosen assets in excess of the benchmark and a risk free rate. The difficulty with this is that you would need to automate the selection of the benchmark, which is somewhat subjective and therefore difficult to do.

Since we are mainly interested in backtesting and have the benefit of 20/20 hindsight; it's very easy to select the best assets in the rear view mirror. The choice of assets is therefore not as valuable as a backtest screen, and probably doesn't follow the intent of the volume factor anyways. For now, let's put this one aside and focus on of the trading pattern.

Trading Pattern

A simple approach to look at the value of trading is to compare desired metrics between an active trading strategy and a buy-and-hold strategy with the same assets held at the average allocation from the trading model.

A simple example: we trade SPY using the traditional 200 day moving average, if SPY is above its 200MA, we buy; if it's below; we sell and buy IEF (10 year treasuries). The following backtest shows results from 1/1/2009 to 12/31/2018. From the overview, and with rf = 0; the RoMaD = 10.757%/17.3% = 62%. From the results report, the average allocation is 79% SPY and 21% IEF. I can't find the Sortino anywhere on the output, but let's assume the negative deviation is 9%; and the Sortino ratio is therefore 10.757/9 = 1.2.

Now we take the average allocation from the active trading strategy (79% SPY and 21% IEF) and see how things look with a buy-and-hold strategy. We get a RoMaD of 11.454%/22.1% = 52%; and an estimated Sortino of 11.454/10.5 = 1.09.

We now need to determine the value of active trading. We'll ignore the tax impact of turning over the portfolio since our target customers are hedge funds and not individual investors.

Active Strategy:

• Annual Return: 10.76%
• Negative Std Dev: 9%
• Max Draw Down: 17.3%

• Annual Return: 11.45%
• Negative Std Dev: 10.5%
• Max Draw Down: 22.1%

The idea is to understand what risk-adjusted value is produced by active trading. The Modigliani risk-adjusted performance looks at the expected return of oe investment if it is scaled to the same risk as another investment. For example, if we look at the expected return of the active trading for the same negative standard deviation of the passive approach, we get 10.76% * 10.5% / 9% = 12.5%. In other words, if we took the same level of risk as the passive approach, we would get an expected return of 12.5% on the active approach, or a 1.05% higher return.

This represents the change in the expected return generated by the trading strategy for the same level of risk as the buy-and-hold strategy. We can do the same thing for the RoMaD; the only difference is the risk metric (max DD instead of neg SD). In the example above, we would get an expected return of 10.76% * 22.1%/17.3% = 13.74%, or a 2.29% higher expected return from the active trading strategy for the same maximum draw down.

The fitness score could conceivably be equal to the average of these two changes in expectation, in the example this would be (1.05%+2.29%)/2 = 1.67%. If this is greater than a threshold (1%?) then the algorithm makes the alpha stream, otherwise it is rejected.

Pros:

• Measures the risk-adjusted value of the trading activity, and not the underlying assets

  • Think of this as a “trading alpha” – the return that is generated by trading

  • The growth of the underlying assets is independent from the trading alpha, so buy AAPL in 1998 would have a trading alpha of 0

• Compares risks at the average level (Sortino) and the extreme (RoMaD), like the original metric

• Works for longer (say monthly) or shorter term (HFT) trading strategies

  • May favor more frequent trading because there are more opportunities for the trading alpha to develop if there are more trades

Cons:

• Requires post-run calculations (buy-and-hold backtest)

• More difficult to test systematically and therefore to know where a good threshold might be

  • However, since this measures an alpha (annual return), a threshold based on judgment from QC, such as 1%, is justifiable

• Doesn't look at absolute returns but rather the additional returns generated by trading

Hey Tristan,

Couple of thoughts regarding your proposed approach:

• Unless I'm mistaken, this is just comparing the Sortino of alpa with Sortino of benchmark (where benchmark is average holding weight of all assets), and saying the Alpha's has to be at least some ratio of the benchmark's (>1.01)?
• I see this working with a very specific set of strategies - namely long-only equity strategies. What about long/short strategies or even market-neutral ones, where your benchmark might very well end up having negative returns or zero weights, meaning your score would be wildly inflated compared to a traditional long-only score? What about other asset classes?

Douglas,

Good points.  Here are some thoughts:

• The ratio of Sortinos is a bit different because the result would be dimensionless.  From the prior example, the Sortinos are 1.2 for the active strategy and 1.09 for the passive.  If we compared the two we would get 1.2/1.09 = 1.1.  I prefer calculating a trading alpha instead because it gives you a percent return, something that is easy to interpret
• I'm not sure why this couldn't be used with long-short strategies or alternative assets.  For a long-short, the benchmark would have negative allocation for the short assets, i.e. you go short and stay short for the duration of the backtest.  Also, I don't know how this would get wildly inflated.  For this to go wild, the risk measure (e.g. max draw down) of the passive strategy would have to be much larger than that of the active strategy.  The formula is simply:

trading alpha = ra * sp/sa - rp

where:

• ra = annual return of active strategy
• rp = annual return of passive strategy
• sa = risk measure, e.g. negative standard deviation or max draw down, of active strategy
• sp = risk measure of passive strategy 

Hi Tristan - I see now that my approximation to ratios of Sortinos isn't quite what you meant, my apologies. I can't quite see a case where the measure would get wildly inflated.

Again, however, I get the impression you're thinking in terms of equities, where indeed a long/short portfolio would typically long some companies and short others, thus ending up with a meaningful measure.

Let's say instead I'm trading a basket of G10 currencies and my active strategy is such that the average weights over the period is 0% across all currencies. rp = 0, sp = 0 - giving a measure of 0 even though the strategy might be profitable.

My point is that the concept of a benchmark is complicated when you start to look outside of equities, where assets often don't have a positive drift and instead act in a more mean-reverting fashion. That's before we even start to talk about options, which brings a wholly different set of problems.

Just throwing this into the discussion: our aim is to remove our human bias so ideally, the fitness function should be automated. If we need to manually construct representative passive portfolios it will induce delays and costs in the evaluation process. If "the community" on a whole has factors they believe should be considered please also sanity check it can be automated!

Jared,

The intent was to automate the benchmark, which could've been done relatively easily, so that an alpha that measures the value of the trading activity could be determined.  However, as Douglas points out (thanks Douglas), this method is not universally applicable, so it's value may be limited.

Jared,

Would you consider lowering the threshold on the trade factor like you mentioned before?  Maybe requiring full portfolio turnover every month to get a score of 100%?  That would mean average turnover of 10% per trading day under the current metric.

Thanks.

I have a suggestion. For leading us to improve Alpha Ranking, the system should present base performance correlating with Alpha Ranking in Overview page in backtest. Such as backtest year, daily portfolio turnover, annual return, max drawdown, annual std of negative return. Now  backtest year, daily portfolio turnover, annual std of negative return are absent.

Hi Li,

Thanks for your suggestion, we will take it into consideration.

Thank you 

I have concerns with Sortino and RoMaD. My model does not do HFT but it takes on average 282 positions per day on the mini ES contract against an average of 19 in live trading. On a 5 year backtest with a resolution. .Tick, the results are abnormally high but the backtest fails because of the valuation mode of the Sortino and RoMaD due to the high volume. I found that limit orders are instantly executed if the price is hit. I also noted orders executed at prices that the ES has never reached since its creation. For those who do HFT, is a 3 month backtest enough to submit an HFT model in the Alpha stream market?

Hi Aliou,

Although we no longer use the Fitness Function proposed in the research notebook at the beginning of this thread, 3-months is, unfortunately, not a long enough backtest to submit. Alphas need to demonstrate consistent performance over a long period of time. Funds want to be able to see its performance in a variety of market regimes and a 3-month backtest isn't sufficient for this. You can find details about the Alpha submission criteria here.