Hi,

It took some time, more than a year to understand the problem I've facing.
My post includes some analysis of QC data against broker (oanda) data with timezone handlings. 
 

First of all some background information about the broker API:

OANDA API uses dates in RFC3339 or unix time representation. Here is a sample request/response:

/v3/instruments/EUR_USD/candles?price=BA&from=2019-04-07T21:02:00.00000000Z&granularity=M1&count=1 {    "instrument": "EUR_USD",    "granularity": "M1",    "candles": [   {       "complete": true,       "volume": 2,       "time": "2019-04-07T21:02:00.000000000Z",       "bid":       {          "o": "1.12125",          "h": "1.12125",          "l": "1.12125",          "c": "1.12125"       },       "ask":       {          "o": "1.12225",          "h": "1.12225",          "l": "1.12225",          "c": "1.12225"       }    }] }


Then, some here is my understanding about lean datetime handling:

  • The SetTimeZone can set used to time zone, but default time is set to New York timezone.
  • QCAlgorithm.Time has not "timezone markup", but it represents the algorithm timezone. This "markup" is important to convert the Time to other timezones. That means that functions such as ToUniversalTime() does not work because it lacks this "markup".
  • Nevertheless Lean provide a means to to get the UTC time: QCAlgorithm.UtcTime
  • The timestamp of the data uses the Exchange.TimeZone, so to convert into UTC we must use
  • data[symbol].Time.ConvertToUtc(Portfolio.Securities[symbol].Exchange.TimeZone)

Question, will the machine timezone influence the algorithm? (I dont think so, but for the moment I have no evidence)

Now the problem I've encountered is related to data not being fed into the algorithm on sundays.

Note however, that data prices are exactly the same as the broker prices when the data is fed. The only issue I encounter is the lack of data on sundays.

Here is how to reproduce:

Use EUR_USD instrument and start algo on sunday.

For instance, the EURUSD market hours starts on the 7 april 17:00 Exchange Time or 21:00 UTC time. In the below logs you can see that the data provided by QC starts from 23:OO UTC time, and not from 21:00 as expected.

Please find below the logs of the data for the algo Run between 2019-04-07 => 2019-04-08
 

2019-04-08 00:01:00 oanda 1.12199 UTC 4/7/2019 11:01:00 PM Exchange 4/7/2019 7:01:00 PM 2019-04-08 00:01:00 leandata 1.12195 UTC 4/7/2019 11:00:00 PM Exchange 4/7/2019 7:00:00 PM 2019-04-08 00:01:00 time 4/8/2019 12:01:00 AM Unix 1554681660 FakeUTC 4/8/2019 12:01:00 AM UTC 4/7/2019 11:01:00 PM 2019-04-08 00:02:00 oanda 1.12194 UTC 4/7/2019 11:02:00 PM Exchange 4/7/2019 7:02:00 PM 2019-04-08 00:02:00 leandata 1.12199 UTC 4/7/2019 11:01:00 PM Exchange 4/7/2019 7:01:00 PM 2019-04-08 00:02:00 time 4/8/2019 12:02:00 AM Unix 1554681720 FakeUTC 4/8/2019 12:02:00 AM UTC 4/7/2019 11:02:00 PM 2019-04-08 00:03:00 oanda 1.12194 UTC 4/7/2019 11:04:00 PM Exchange 4/7/2019 7:04:00 PM 2019-04-08 00:03:00 leandata 1.12194 UTC 4/7/2019 11:02:00 PM Exchange 4/7/2019 7:02:00 PM 2019-04-08 00:03:00 time 4/8/2019 12:03:00 AM Unix 1554681780 FakeUTC 4/8/2019 12:03:00 AM UTC 4/7/2019 11:03:00 PM 2019-04-08 00:04:00 oanda 1.12194 UTC 4/7/2019 11:04:00 PM Exchange 4/7/2019 7:04:00 PM

Here, you can see prices the broker provides data since the market openning.

2019-04-08 00:01:00 oanda 1.12225 UTC 4/7/2019 9:02:00 PM Exchange 4/7/2019 5:02:00 PM 2019-04-08 00:01:00 time 4/8/2019 12:01:00 AM Unix 1554681660 FakeUTC 4/8/2019 12:01:00 AM UTC 4/7/2019 11:01:00 PM 2019-04-08 00:02:00 oanda 1.12225 UTC 4/7/2019 9:02:00 PM Exchange 4/7/2019 5:02:00 PM 2019-04-08 00:02:00 time 4/8/2019 12:02:00 AM Unix 1554681720 FakeUTC 4/8/2019 12:02:00 AM UTC 4/7/2019 11:02:00 PM 2019-04-08 00:03:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM 2019-04-08 00:03:00 time 4/8/2019 12:03:00 AM Unix 1554681780 FakeUTC 4/8/2019 12:03:00 AM UTC 4/7/2019 11:03:00 PM 2019-04-08 00:04:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM 2019-04-08 00:04:00 time 4/8/2019 12:04:00 AM Unix 1554681840 FakeUTC 4/8/2019 12:04:00 AM UTC 4/7/2019 11:04:00 PM 2019-04-08 00:05:00 oanda 1.12202 UTC 4/7/2019 9:05:00 PM Exchange 4/7/2019 5:05:00 PM 2019-04-08 00:05:00 time 4/8/2019 12:05:00 AM Unix 1554681900 FakeUTC 4/8/2019 12:05:00 AM UTC 4/7/2019 11:05:00 PM 2019-04-08 00:06:00 oanda 1.12211 UTC 4/7/2019 9:06:00 PM Exchange 4/7/2019 5:06:00 PM 2019-04-08 00:06:00 time 4/8/2019 12:06:00 AM Unix 1554681960 FakeUTC 4/8/2019 12:06:00 AM UTC 4/7/2019 11:06:00 PM 2019-04-08 00:07:00 oanda 1.12206 UTC 4/7/2019 9:07:00 PM Exchange 4/7/2019 5:07:00 PM 2019-04-08 00:07:00 time 4/8/2019 12:07:00 AM Unix 1554682020 FakeUTC 4/8/2019 12:07:00 AM UTC 4/7/2019 11:07:00 PM


Please refer to the below backtest.

 

Author