Realistic backtesting question by Nik milev

Nik milev INVESTOR

| Reward Discussion

HI guys,

I ran some back tests and live trading (IB) for the same ago and observed differences. I would like to make the back testing more realistic. Here is what happens:

Assume I run on 1 sec resolution and current price of the "abc" security is $10.00 and I have custom slippage e.g. 0.03%

Backtesting: when i send limit order to buy at $10.05, the fill price is $10.00 - it seems that it takes current closing price, since it is within the buy limit.

Live run: it fills at e.g. $10.04

Is it possible in backtesting to change the orders fills to happen on the next bar, e.g. to set this somehow during initialization ? This will make it a bit more realistic.

Other ideas a very welcome!

Thanks

Nik

Update Backtest

Nik milev

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Stefano Raggi

Pro ,

Hi Nik,
just a couple of notes to clarify since you mentioned limit orders:

The backtesting engine only fills Market orders on the current bar, other order types are always filled on the next bar.
Assuming you are using the default fill model, ImmediateFillModel, slippage is never applied to limit order fills.

This is the logic for limit buy order fills in this model:

if (prices.Low < order.LimitPrice)

{

    //Set order fill:

    fill.Status = OrderStatus.Filled;

    // fill at the worse price this bar or the limit price, this allows far out of the money limits

    // to be executed properly

    fill.FillPrice = Math.Min(prices.High, order.LimitPrice);

}

from your example it seems the High of the next bar ($10.00) was actually lower than the order limit price ($10.05).

A few suggestions:

maybe you could log the current and next bar OHLC to verify if it's not a data issue
if the data is correct, you might want to write a custom fill model, deriving from ImmediateFillModel and overriding only the LimitFill method
another option could be to use Tick resolution -- the backtest will be slower, but results should be more realistic

Nik milev

103.9k Pro ,

HI Stefano,

Thanks for the quick response!

So, since the limit order fills on next bar, you think next bar is $10.

How about when during next bar/second there was no trade?

I would guess that the previous bar close of $10.00 would be carried over

and will cause the fill to happen on next bar which is in effect unchanged

previous bar, due to no trades.

What do you think?

As you said may be using tick is better option

Thanks

Nik

Stefano Raggi

Pro ,

Hi Nik,
the limit order fill is not guaranteed to happen on the next bar, depending on the price action it could happen later:
e.g. if a limit buy order is at $10 and the Low of the next bar is at $10.05 there will be no fill and the order will remain active.

If there are no trades in the resolution period, the bar will be copied forward but should not be used for fills, that would be a huge LEAN bug :)

Nik milev

103.9k Pro ,

HI Stefano,

Thanks for the clarification that the artificially copied bar will not be

used for fills, I did not know this!

I will try to create a clean test to better see what happens.

Thanks for the support!

Nik

Nik milev

103.9k Pro ,

Hi Stefano,

I cleaned a little bit the test and ran it again. This time I used TYD and DRN - both trade very few shares. Basically I buy every 15th minune and sell every 33rd minute (sending limit orders). If after 1 minute the orders are not filled, they are canceled and all is liquidated.

I put limit order to enter few cents above the current price and to exit few cents above the current price.

As I see it, it alwas enters and almost never exits - check TYD. So it seems that it uses next bar to enter, even if there is no trade.

Unless I have a mistake in the code which I cant see, since it is hard to debug, it seems to me that e.g. the algo uses next bar of TYD to enter and caries over TYD bar to next second - prices and volume seem suspiciously the same from 10:15 till 11:30 and etc.

Your opinion is greately appreciated.

Thanks

Nik

9/17/2018 10:15:00 AM BUY order1 sent, .status=New, limit= 40.090, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 10:15:00 AM BUY order2 sent, .status=New, limit= 23.430, curPrice=23.410, high=23.410,low=23.410, vol=300.000
9/17/2018 10:15:00 AM, Will print next bar
9/17/2018 10:15:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 10:15:01 AM, barPairNow[1]=23.410, high=23.410, low=23.410, vol=300.000
9/17/2018 10:15:01 AM Entry Orders Executed , order1.FilledPrice=40.07, order2.FilledProce=23.41
9/17/2018 10:33:00 AM EXIT order1 sent, .status=New, limit= 40.080, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 10:33:00 AM EXIT order2 sent, .status=New, limit= 23.350, curPrice=23.360, high=23.360,low=23.360, vol=500.000
9/17/2018 10:33:00 AM Exit orders sent
9/17/2018 10:33:00 AM, Will print next bar
9/17/2018 10:33:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 10:33:01 AM, barPairNow[1]=23.360, high=23.360, low=23.360, vol=500.000
9/17/2018 10:33:01 AM exit order1 canceled
9/17/2018 10:33:02 AM HandleState_PendingOrderExitCancelationSent - Liquidated ALL exit orders if any
9/17/2018 11:15:00 AM BUY order1 sent, .status=New, limit= 40.090, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 11:15:00 AM BUY order2 sent, .status=New, limit= 23.170, curPrice=23.150, high=23.150,low=23.150, vol=5.000
9/17/2018 11:15:00 AM, Will print next bar
9/17/2018 11:15:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 11:15:01 AM, barPairNow[1]=23.150, high=23.150, low=23.150, vol=5.000
9/17/2018 11:15:01 AM Entry Orders Executed , order1.FilledPrice=40.07, order2.FilledProce=23.15
9/17/2018 11:33:00 AM EXIT order1 sent, .status=New, limit= 40.080, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 11:33:00 AM EXIT order2 sent, .status=New, limit= 23.180, curPrice=23.190, high=23.190,low=23.190, vol=406.000
9/17/2018 11:33:00 AM Exit orders sent
9/17/2018 11:33:00 AM, Will print next bar
9/17/2018 11:33:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 11:33:01 AM, barPairNow[1]=23.190, high=23.190, low=23.190, vol=406.000
9/17/2018 11:33:01 AM exit order1 canceled
9/17/2018 11:33:02 AM HandleState_PendingOrderExitCancelationSent - Liquidated ALL exit orders if any

Stefano Raggi

Pro ,

Hi Nik,

I checked the data files for TYD, these are the first two (second resolution) bars in the data file for 9/17:

34200000,400700,400700,400700,400700,200 // 09:30:00

51033000,399800,399800,399800,399800,200 // 14:10:33

Since fill forward is enabled by default, you will receive the first bar in OnData multiple times (from 9:30 until 14:10 when there was a new trade), this explains why you are seeing the same prices (and volume) repeated in the logs.
Regarding the limit order fills in backtesting, while it is true that limit order fills are always filled on the next bar, the prices used to determine if a fill occurred are from the last bar received (which can actually be a previous bar copied forward -- apologies for the confusion I might have caused in my previous post).
In equity backtesting we don't have bid/ask quotes, so we can only use bars created from trade ticks.

After running your test algorithm locally, I verified that the output is expected, this is the sequence of events:

at 9:30 the first bar comes in with all OHLC values equal to 40.07
at 10:15:00 the buy order for TYD was submitted (limit price = 40.09)
at 10:15:01 the buy order was filled at 40.07
- the fill price was lower because in the current bar 40.07 is the highest value traded

at 10:33:00 the sell order was submitted (limit price = 40.08)
- this order was not filled because the limit price was higher than the High of current bar (40.07)

Sadly when the first order was submitted at 10:15 the only price data available was a bar from 45 minutes earlier.

In live mode you would get real time quote prices, so differences in fill prices are expected.

In this case, even using Tick resolution wouldn't help, there were no trades from 9:30 to 14:10.

Returning to the original question on how to make backtesting more realistic, I think maybe a custom fill model could try to estimate the bid/ask prices and handle limit order fills more accurately (modeling thinly traded equities like TYD might be hard though).

Jared Broad

Pro ,

Thanks for raising this case Nik; its an interesting extreme. We'll review the fill models and make sure they'll behave as expected - filling on the next appropriate bar open/high/low/close price based on the order type.

Nik milev

103.9k Pro ,

HI Jared and Stefano,

Thanks for looking at this issue!

Filling on last and carried over old bar creates problem and skews test

especially for not so traded securities, and when we use higher resolutions

( e.g. seconds).

Few things come to my mind I would like to share with you:

1. It would be beneficial for me as a developer to somwhow know if current

bar is built from real data or just carried over.

There are several ways to do this, and one way would be to add a property

to TradeBar, showing the time of the last ticker/trade. This way, I can see

that the closing price is not in e.g. minute/second (my resolution) and

will act accordingly.

2. Another thing to do along with above idea (to notify the algo that the

bar is artifficial) is to provide several fill modes.

In case when no bar/trade is in the next resolution interval, an

approximation artifficial bar can be created, based on the last trade and

future trade. This bar prices will be approximation - the close time is to

the future trade, the better its prices match.

I guess same can be done for volume (or always set volume=0) for

artifficial bar.

Currently, trading seconds resolution for average volume trading securities

artificially boost the returns.

I am looking forward to know more about your decision and implementation!

Thanks

Nik

Stefano Raggi

Pro ,

Hi Nik,
we totally agree that order fills for limit (and stop/stoplimit) orders should never be emitted if there is no new data,
there is a proposed fix here (currently under review):

https://github.com/QuantConnect/Lean/pull/2548

For the volume in fill-forward bars being copied over from previous bars, this was definitely a bug and has already been fixed and merged yesterday:

https://github.com/QuantConnect/Lean/pull/2540

Regarding your two questions:
1. In OnData, to know if a bar is real or fill-forward, you can use the BaseData.IsFillForward property:

var isFillForward = bar.IsFillForward;

2. There is no easy way to generate artificial bars in LEAN for any symbol, all existing models should work with the actual (real) trade data.

Hopefully the two bug fixes above will improve the accuracy of your backtesting results.

Nik milev INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

Community Ambassadors

Earn free QuantConnect cloud credits by helping others, each week we airdrop credits to the most active community members.

233,126 Quants.

Online Now

Stay Connected