Same code, different backtest results

I've seen this occur again in another backtest. Different results for the same code. The trades are different as well.

I found that in one backtest, the pricing seems incorrect, while in the other, the pricing is correct. 

Specifically, in the ‘correct’ backtest, my algorithm executes 

but in the ‘incorrect’ backtest, the algorithm executes

I'm saying correct/incorrect based on the fact that for the second backtest, the value $7.90 is far from the price in the following orders for UVXY. 

It seems like a data issue, but even so I would like to know why it occurs randomly.

Hi AK M,

Normally (without bugs), the same code on the same assets for the same period of time should be deterministic, UNLESS the strategy/algo itself is not deterministic.

In other words, if an algo is deterministic, by definition, it means that the same inputs are going to produce the same outputs.

From what you are writing, it looks like an error in the data. But I don't know why one execution triggers it and not the other. Looks like the algo you are using is not deterministic in some way… Or you are hitting some bug in the platform on one of the two executions.

Fred

Yes, I can confirm that my algorithm has no random elements. It doesn't use ML or anything that requires a random initial state either. 

These changes did happen after I started using the SpreadExecutionModel(0.001) but I have no idea if that is indeed the root cause.

A closer look at the source doesn't reveal any random elements: 

I also requested a backtest report, just in case it was a UI rendering issue. But the backtest report is also erroneous. 

Well, without the code, it is difficult, or some would say impossible 😊, to say.

However, note that nondeterminism is not only a question of explicit randomness used in the algo. That “randomness”, which yes is a nice intuitive way to grasp nondeterminism, can be introduced by many many other ways. Refer to the most succinct and incomplete overview of nondeterminism I've seen (for the first time now 😊), https://en.wikipedia.org/wiki/Nondeterministic_algorithm, for a few other means. The most common way, I'd say without real backing data, not in the list, is a non-crashing bug in software. That bug can easily be triggered only on certain execution paths and can lead to different states in the program without those states being invalid enough to make it crash but still leading to different outputs. That bug could be in the algo itself, but also anywhere in the stack: LEAN, CLR/Python runtime, OS, drivers, firmware, hardware…

And there are other reasons too. Another classic, and that one is listed, is threads/true parallelism. LEAN is multi-threaded and most certainly runs on multi-core computers (true parallelism) - in backtest, live, at your place, in the cloud.

Fred