Back

Data gap issue

Hi,

It took some time, more than a year to understand the problem I've facing.
My post includes some analysis of QC data against broker (oanda) data with timezone handlings. 
 

First of all some background information about the broker API:

OANDA API uses dates in RFC3339 or unix time representation. Here is a sample request/response:

/v3/instruments/EUR_USD/candles?price=BA&from=2019-04-07T21:02:00.00000000Z&granularity=M1&count=1
{
   "instrument": "EUR_USD",
   "granularity": "M1",
   "candles": [   {
      "complete": true,
      "volume": 2,
      "time": "2019-04-07T21:02:00.000000000Z",
      "bid":       {
         "o": "1.12125",
         "h": "1.12125",
         "l": "1.12125",
         "c": "1.12125"
      },
      "ask":       {
         "o": "1.12225",
         "h": "1.12225",
         "l": "1.12225",
         "c": "1.12225"
      }
   }]
}


Then, some here is my understanding about lean datetime handling:

  • The SetTimeZone can set used to time zone, but default time is set to New York timezone.
  • QCAlgorithm.Time has not "timezone markup", but it represents the algorithm timezone. This "markup" is important to convert the Time to other timezones. That means that functions such as ToUniversalTime() does not work because it lacks this "markup".
  • Nevertheless Lean provide a means to to get the UTC time: QCAlgorithm.UtcTime
  • The timestamp of the data uses the Exchange.TimeZone, so to convert into UTC we must use
  • data[symbol].Time.ConvertToUtc(Portfolio.Securities[symbol].Exchange.TimeZone)

Question, will the machine timezone influence the algorithm? (I dont think so, but for the moment I have no evidence)

Now the problem I've encountered is related to data not being fed into the algorithm on sundays.

Note however, that data prices are exactly the same as the broker prices when the data is fed. The only issue I encounter is the lack of data on sundays.

Here is how to reproduce:

Use EUR_USD instrument and start algo on sunday.

For instance, the EURUSD market hours starts on the 7 april 17:00 Exchange Time or 21:00 UTC time. In the below logs you can see that the data provided by QC starts from 23:OO UTC time, and not from 21:00 as expected.

Please find below the logs of the data for the algo Run between 2019-04-07 => 2019-04-08
 

2019-04-08 00:01:00 oanda 1.12199 UTC 4/7/2019 11:01:00 PM Exchange 4/7/2019 7:01:00 PM
2019-04-08 00:01:00 leandata 1.12195 UTC 4/7/2019 11:00:00 PM Exchange 4/7/2019 7:00:00 PM
2019-04-08 00:01:00 time 4/8/2019 12:01:00 AM Unix 1554681660 FakeUTC 4/8/2019 12:01:00 AM UTC 4/7/2019 11:01:00 PM
2019-04-08 00:02:00 oanda 1.12194 UTC 4/7/2019 11:02:00 PM Exchange 4/7/2019 7:02:00 PM
2019-04-08 00:02:00 leandata 1.12199 UTC 4/7/2019 11:01:00 PM Exchange 4/7/2019 7:01:00 PM
2019-04-08 00:02:00 time 4/8/2019 12:02:00 AM Unix 1554681720 FakeUTC 4/8/2019 12:02:00 AM UTC 4/7/2019 11:02:00 PM
2019-04-08 00:03:00 oanda 1.12194 UTC 4/7/2019 11:04:00 PM Exchange 4/7/2019 7:04:00 PM
2019-04-08 00:03:00 leandata 1.12194 UTC 4/7/2019 11:02:00 PM Exchange 4/7/2019 7:02:00 PM
2019-04-08 00:03:00 time 4/8/2019 12:03:00 AM Unix 1554681780 FakeUTC 4/8/2019 12:03:00 AM UTC 4/7/2019 11:03:00 PM
2019-04-08 00:04:00 oanda 1.12194 UTC 4/7/2019 11:04:00 PM Exchange 4/7/2019 7:04:00 PM

Here, you can see prices the broker provides data since the market openning.

2019-04-08 00:01:00 oanda 1.12225 UTC 4/7/2019 9:02:00 PM Exchange 4/7/2019 5:02:00 PM
2019-04-08 00:01:00 time 4/8/2019 12:01:00 AM Unix 1554681660 FakeUTC 4/8/2019 12:01:00 AM UTC 4/7/2019 11:01:00 PM
2019-04-08 00:02:00 oanda 1.12225 UTC 4/7/2019 9:02:00 PM Exchange 4/7/2019 5:02:00 PM
2019-04-08 00:02:00 time 4/8/2019 12:02:00 AM Unix 1554681720 FakeUTC 4/8/2019 12:02:00 AM UTC 4/7/2019 11:02:00 PM
2019-04-08 00:03:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM
2019-04-08 00:03:00 time 4/8/2019 12:03:00 AM Unix 1554681780 FakeUTC 4/8/2019 12:03:00 AM UTC 4/7/2019 11:03:00 PM
2019-04-08 00:04:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM
2019-04-08 00:04:00 time 4/8/2019 12:04:00 AM Unix 1554681840 FakeUTC 4/8/2019 12:04:00 AM UTC 4/7/2019 11:04:00 PM
2019-04-08 00:05:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM
2019-04-08 00:05:00 time 4/8/2019 12:05:00 AM Unix 1554681900 FakeUTC 4/8/2019 12:05:00 AM UTC 4/7/2019 11:05:00 PM
2019-04-08 00:06:00 oanda 1.12211 UTC 4/7/2019 9:06:00 PM Exchange 4/7/2019 5:06:00 PM
2019-04-08 00:06:00 time 4/8/2019 12:06:00 AM Unix 1554681960 FakeUTC 4/8/2019 12:06:00 AM UTC 4/7/2019 11:06:00 PM
2019-04-08 00:07:00 oanda 1.12206 UTC 4/7/2019 9:07:00 PM Exchange 4/7/2019 5:07:00 PM
2019-04-08 00:07:00 time 4/8/2019 12:07:00 AM Unix 1554682020 FakeUTC 4/8/2019 12:07:00 AM UTC 4/7/2019 11:07:00 PM


Please refer to the below backtest.

 

Update Backtest








0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


In addition to issues on backtests, I've found there are also issues in live trading. See the below logs: data retrieved from Oanda vs data obtained from Lean

I'm printing Ask close prices, please not the differences

 

019-04-11 13:51:00 : oanda 1.12671 UTC 4/11/2019 1:50:00 PM Exchange 4/11/2019 9:50:00 AM
2019-04-11 13:51:00 : leandata 1.12683 UTC 4/11/2019 1:50:00 PM Exchange 4/11/2019 9:50:00 AM
2019-04-11 13:52:00 : oanda 1.12662 UTC 4/11/2019 1:51:00 PM Exchange 4/11/2019 9:51:00 AM
2019-04-11 13:52:00 : leandata 1.12674 UTC 4/11/2019 1:51:00 PM Exchange 4/11/2019 9:51:00 AM
2019-04-11 13:53:00 : oanda 1.12679 UTC 4/11/2019 1:52:00 PM Exchange 4/11/2019 9:52:00 AM
2019-04-11 13:53:00 : leandata 1.12691 UTC 4/11/2019 1:52:00 PM Exchange 4/11/2019 9:52:00 AM
2019-04-11 13:54:00 : oanda 1.12680 UTC 4/11/2019 1:53:00 PM Exchange 4/11/2019 9:53:00 AM
2019-04-11 13:54:00 : leandata 1.12692 UTC 4/11/2019 1:53:00 PM Exchange 4/11/2019 9:53:00 AM

1:50PM UTC

{
"instrument": "EUR_USD",
"granularity": "M1",
"candles": [ {
"complete": true,
"volume": 72,
"time": "2019-04-11T13:50:00.000000000Z",
"bid": {
"o": "1.12665",
"h": "1.12672",
"l": "1.12658",
"c": "1.12660"
},
"ask": {
"o": "1.12677",
"h": "1.12685",
"l": "1.12669",
"c": "1.12671"
}
}]
}

1:51 PM UTC

{
   "instrument": "EUR_USD",
   "granularity": "M1",
   "candles": [   {
      "complete": true,
      "volume": 48,
      "time": "2019-04-11T13:51:00.000000000Z",
      "bid":       {
         "o": "1.12661",
         "h": "1.12665",
         "l": "1.12646",
         "c": "1.12650"
      },
      "ask":       {
         "o": "1.12672",
         "h": "1.12677",
         "l": "1.12657",
         "c": "1.12662"
      }
   }]
}

 

0

Some more data for live trading.

2019-04-11 14:56:00 : oanda BidOpen 1.12739 AskOpen 1.12751 BidClose 1.12743 AskClose 1.12753 UTC 4/11/2019 2:55:00 PM Exchange 4/11/2019 10:55:00 AM
2019-04-11 14:56:00 : leandata BidOpen 1.12729 AskOpen 1.12766 BidClose 1.12731 AskClose 1.12765 UTC 4/11/2019 2:55:00 PM Exchange 4/11/2019 10:55:00 AM
2019-04-11 14:57:00 : oanda BidOpen 1.12741 AskOpen 1.12752 BidClose 1.12745 AskClose 1.12758 UTC 4/11/2019 2:56:00 PM Exchange 4/11/2019 10:56:00 AM
2019-04-11 14:57:00 : leandata BidOpen 1.12729 AskOpen 1.12764 BidClose 1.12733 AskClose 1.1277 UTC 4/11/2019 2:56:00 PM Exchange 4/11/2019 10:56:00 AM
2019-04-11 14:58:00 : oanda BidOpen 1.12745 AskOpen 1.12756 BidClose 1.12729 AskClose 1.12742 UTC 4/11/2019 2:57:00 PM Exchange 4/11/2019 10:57:00 AM
2019-04-11 14:58:00 : leandata BidOpen 1.12733 AskOpen 1.1277 BidClose 1.12717 AskClose 1.12754 UTC 4/11/2019 2:57:00 PM Exchange 4/11/2019 10:57:00 AM
2019-04-11 14:59:00 : oanda BidOpen 1.12731 AskOpen 1.12742 BidClose 1.12736 AskClose 1.12748 UTC 4/11/2019 2:58:00 PM Exchange 4/11/2019 10:58:00 AM
2019-04-11 14:59:00 : leandata BidOpen 1.12719 AskOpen 1.12754 BidClose 1.12724 AskClose 1.1276 UTC 4/11/2019 2:58:00 PM Exchange 4/11/2019 10:58:00 AM

 

0

Forget about live trading, prices at live trading are reflecting oanda API. The issue is with backtesting data

0

I am seeing the same on my side - I am neither unable to reproduce live trading results in a backtest nor able to match historical csv data to what I'm seeing directly from Oanda.

Also, I'd like to highlight that in a live deployment (on the terminal), an Oanda daily bar closes at 0AM EDT/EST when it should be UTC (as per the data timezone).

Hopefully we can look into these issues and come to a resolution.

0

Douglas, I use my OandaDataProvider to replace the standard Lean downloader. It fixes the issue when I run Lean. I will stick to it for a while.

https://github.com/ebengtso/quant/tree/master/OandaDataProvider

 

0

Update Backtest





0

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.


Loading...

This discussion is closed