This thread is a continuation of the discussion in the "Intersection of ROC comparison using OUT_DAY approach" thread. I argued there, that a forum dedicated to quant finance should not just be about producing backtests, and chasing hypothetical returns, Sharpe ratio's and the like through curve fitting. While many of such threads start with an interesting premise, they generally devolve into a rat race for the highest return with many apparently oblivious to the dangers of such an approach. Worse yet, with all of these threads in the "interesting" category, it gives new members the impression, that this is what quant finance is about. In my view it is not. Backtests are a useful tool for testing hypotheses, but they should be approached with extreme caution, since merely the process of producing dozens of backtests significantly increases the likelihood of making a false discovery.  

Additionally, there has been quite some pushback from those that engage in what I'm sorry to say borders on pseudoscience, particulary when good scientific practices are dismissed off-hand by those that have little or no experience in the field of science. The fact that participating in the echo chamber is considered "constructive comments", while offering a professional, and scientific perspective, and discussing the merits of the processes used to generate, and evaluate backtests is viewed as a "glass half full" mentality is a major problem in strategy development. It is not easy to evaluate startegies, and it's even more difficult to evaluate your own work in an unbiased, objective manner. It is actually much easier to produce a seemingly good in-sample backtest, and to convince others you've struck gold, than it is to convince others that what glitters (more often than not) isn't gold. Many people look at a backtest, like they look at the top image in the below figure, and conclude square A is a darker shade of gray than square B, whereas in reality it is an optical illusion, and both squares are the same color, as demonstrated in the bottom image:


The objective of this thread to the discussions on this forum is not to provide people with another equivalent of the top image, but to highlight the tools, that can be used to break the illusion, as is done in the bottom picture, such that we don't waste a lot of time chasing ghosts, but use all of the tools available to us, such that we can state with confidence, that we've made an actual discovery, and when we make an improvement to a strategy, that we can state with confidence, that it is an actual improvement. I invite everyone here to give their perspective on the problem of false discoveries in finance, and in the development of trading strategies in particular.