Back

9 Ways to Get Your Alpha Rejected!

 

Hey everyone!

We know that Alpha Streams can be a bit confusing to new users, and even for experienced users the acceptance/rejection criteria can still be a bit fuzzy. Right now, we're seeing an 80-90% rejection rate, so we've outlined nine of the most common reasons we reject Alpha Streams submissions. Hopefully, this answers some questions you had or clears up any confusion out there. If you have more questions for us, post in the comments below!

  1. Failure to use required API
    • All Alpha Streams submissions are now required to have at least 5-years of backtesting history, which should cover the most recent 5-years up to the present. The results between a submission that doesn't use five years and our test can differ substantially, and so we will reject it so the author can continue to test their model.
    • All Alpha Streams submissions must use the Alpha Streams Brokerage Model. This model is built to model the specific fees, borrowing costs, and other constraints on trading that funds will experience when trading a licensed Alpha. We're continuing to add functionality to this model, so expect some changes in the future.
    • Submissions that use Indicators or Consolidators must use either the SetWarmup or History API. SetWarmup should be used for any Indicators that are specifically assigned to a symbol or registered with the algorithm in the Initialize() method. Any Indicators initialized elsewhere (i.e., in OnSecuritiesChanged) will need to use the History API to warm them up. Using the History or SetWarmup API will ensure that the algorithm is trading under the correct circumstances as soon as it is deployed and will regain state immediately in the case of a restart.
  2. Insights
    • Insights are predictions about the price movement of specific securities and are required in all Alpha Streams submissions. For classic algorithms, one of the Insight constructors must be used and emitted using the EmitInsights API. An Insight must be emitted before any orders are placed. Insights provide funds with information about the predictive power of your models and give them insight into why the orders they see are being placed. For Framework-style algorithms, Insights need to be generated in the Update() method of the Alpha Model. These will then be used in the Portfolio Construction model to build a portfolio.
    • Insights must contain information about the security, the duration of the anticipated price movement, and the direction. Additionally, it is extremely valuable for the Insights to have one or more of the magnitude, confidence, or weighting arguments. These arguments give funds an idea of what the expected movement of a security's price will be, which predictions are particularly relevant to determine optimal capital allocation, and ones that they can act upon with confidence.
  3. Daily Data and Intraday Events
    • We aggregate our Daily data and then passed through the algorithm at 00:00 UTC the day after (i.e., the Daily resolution TradeBar for 2019-10-22 will pass through the algorithm at 2019-10-23 00:00). When using daily data, any operations performed during an intraday event will be using stale data. Using daily data can lead to unexpected results and trades, as well as unrealistic performance. All orders are placed as Market On Open orders, which is often not what people expect when they use Daily data. Not fully understanding Daily data can cause confusion about the behavior of an algorithm and lead to performance that can't be replicated by a fund licensing the Alpha. In general, it is best to use Minute or Hour resolution and a Scheduled Event to perform daily operations to achieve realistic performance.
  4. Open-Source IP
    • We provide the community with lots of example algorithms. The goal of this is to demonstrate how to use the API correctly, incorporate new data sources into their existing algorithms, implement our recommended best practices for Alpha Streams, and more. However, we cannot accept any copies or near-copies of our demonstration algorithms into Alpha Streams. We love to see that our work inspired someone, but each submission must provide sufficient originality that the work can indeed be called the author's own.
  5. External Data Sources
    • When a fund licenses an Alpha, we need to guarantee them that it will perform without error and be of the highest quality possible. To do this, we can't accept any submissions that use an external data source that is not built into Lean (i.e., dropbox files). If we don't directly support the data, then we are unable to guarantee its integrity and that it will be maintained faithfully. We do our best to provide an extensive data library, and we are always working on adding new data sources.
  6. Overfitting
    • Overfitting will doom an algorithm in live trading. To prevent this and try to boost the quality of submissions, we can't accept any Alphas that obviously overfit to data. Overfitting can manifest itself in countless ways, but the most common things we see are
      • Coding of indicator parameters that work for certain hand-picked assets but perhaps not for any others.
      • Using thresholds for indicator values that are hard-coded and have no fundamental theory behind their value.
      • Look-ahead bias, such as hard-coding specific dates to perform specific actions. This can only be done if there is prior knowledge of an event. Look-ahead bias might boost the backtest, but it does not mean the model is fundamentally valuable and can sustain performance during future outliers/extreme events
      • Selection bias, such as picking stocks known ahead of time to perform exceptionally well during specific periods.
        • ETF baskets are common ways to trade, but the more algorithmic your selection process can be (such as using the Coarse/Fine universe selection), the more valuable your model is.
        • Trading just SPY won't provide much value for a fund, but a model that trades various securities, especially across sectors, has substantial value.
    • For more information on common overfitting mistakes, please see our documentation section and this blog post we made about overfitting and the research process.
  7. Slow Recovery
    • Algorithms that fail to recover from a drawdown within 6-months cannot be accepted. Funds understand that all algorithms experience drawdowns, and the smaller, the better, but the recovery time is especially important. An algorithm that is in a sustained drawdown for more than 6-months most likely won't be traded by a fund and would likely be dropped if this occurs in live trading.
  8. Profitability
    • Alphas need to be profitable over the 5+ year backtest. PnL is not the best metric for success or the value of an algorithm, but accepted Alphas must still demonstrate the ability to generate a profit.
  9. Infrequent Insight Generation
    • Algorithms don't need to trade daily or intraday, but the maximum holding period for funds is usually a matter of days or weeks. Anything longer than monthly-rebalancing can't be accepted. Additionally, High-Frequency Trading algorithms might perform in backtesting, but the reality of live borrowing costs means that almost all HFT algorithms fail in live trading. For the competition, algorithms must emit Insights daily or near-daily basis -- for any week in the backtest, there must be at least 5 Insights.
Update Backtest





The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.



0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thanks Jack, this is a great summary and I hope you copy paste it into other documentation sources.

2

Thanks for posting this Jack. It will also help my live trading.  I need to work on overfiting and infrequent insight generation. 

1

You don't mentionned Future markets are not allowed yet for Alpha's submition.

1

Thanks Jack, super insightful guide!

1

Since overfitting was mentioned, are Alphas on the marketplace allowed to rely on machine learning? If they are, how do you determine whether or not they're "obviously overfit"?

2

Hi Krzysztof Steciag ,

Machine Learning is allowed.
We would determine whether there are obviously overfit by the amount of hard-coded parameters.

1

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Thank you. Extremely useful. 

0

Are firms able to modify their leverage for a strategy that they are licensing to set their own risk appetite?

If not, do the participating firms have a preference for:

A. 1x leverage with lower returns (but less risk).

B. Higher leverage to maximise returns (but with higher risk).

 

0

Should a benchmark be set to suit the style of the strategy? If yes, what benchmark would be appropriate to set for a market neutral pairs trading stratety?

Alternately is it best to just use a common equity index like SPY in most cases?

 

0

Is there an example on github which shows the correct usage of emitting insights from a classic algorithm?  

0

Hi Kieran Flynn ,

Good question!

On leverage:
All Alpha Streams submissions must use the Alpha Streams Brokerage Model. It forces 1x leverage.

On benchmark:
It is best to use SPY in most cases.

The correct way to emit insights depends on how the algorithm is processing orders.
If orders are handled by an ExecutionModel (except the default NullExecutionModel), we better collect all insights in a given method in a list and call EmitInsights at the end. If we emit individual Insight, the PortfolioConstructionModel may re-process the same Insight,
If not, the algorithm can call EmitInsights for each individual Insight before placing the order with an "order" method (SetHoldings, Liquidate, MarketOrder, LimitOrder, etc).

0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed