Different behavior for custom data plot using QC in cloud and desktop

I'm guessing, but I'd say it's local time vs UTC.

I found this link searching the inet for  setting timezone on quantconnect. I to am a noob, so I'm very likely wrong :)

Hello Wink, 

Thanks for your comment. 

The problem is I am using the *same* piece of code in both platforms to do the test. I expect I would get the same result from both tests no matter the results are right or wrong. 

But in this case, the result does not agree with each other. That's what I am trying to understand. :)
 

Yea, I agree it would nice if they were the same and the problem may have nothing to do with my timezone guess. We'll have to see what the experts say.

Lean has an extension methods that can be used to make timezone convertions.
In this case, you could use ConvertFromUtc:

 This solution has fixed the mismatch issue for me.

Hello Alexandre, 

Thanks a lot for your guidance. 

First of all, I made some minor changes in your code, see comment below about the changes.

Now with that change, the 'cloud' version of LEAN plot output agrees with the log , they both start at 8:50am 07/26/2016.

Now the problem again is on LEAN desktop version. Timing of the plot does not agrees with the log. In this case, I have the plot starts at 00:50am 07/26/2016 while the log prints 08:50am.

I tried to convet the UTC to other timezone such as NewYork, the disagreement still there. Plot start at 01:50am while the log prints start at 09:50am.

One more interesting thing I find out is that, the plot of LEAN desktop seems to change according to the time zone of setting of my PC.  

For example, when my PC timezone is set to UTC+08,  plot starts at 00:50am while log starts at 08:50am.  

When I change my PC timezone to UTC-5, plot starts at 12:50pm while log starts at 08:50am.

The log seems to give the result I want but the plot seems never have it done right.

One more thing I want to point out , not sure if that relates to this problem, There's no sample of import data until 20 minutes after market open. (i.e. the first data is received at 09:50am Eastern Time. Just want to highlight this in case it is related.

Why this inconsistency happens? Are there any additional setting we need to invoke when we move the code from cloud to desktop?

 Many thanks 
 

anyone has idea on this? I digged into the Community but seems nobody has problem with this.

Not sure if most of the users are using daily data (and hence no need to handle the timezone issue) or I have overlooked/missed some config/settings here. 

Many thanks eveybody

When we convert a string to a DateTime object, using DateTime.ParseExact, the result will take into consideration the operational system timezone (link) and DateTimeKind will be set to Local or Unspecified, unless we specify the style, like you did, using DateTimeStyles.AdjustToUniversal.

However, when we use the ConvertFromUtc method to convert to Chicago Time from UTC, DateTimeKind is set to Unspecified.

Why does the DateTimeKind matter? Because when we plot the data, each data point time coordinate will be converted to UTC by the DateTime.ToUniversalTime method here. If DateTimeKind is not Utc, this method will assume we want to convert local (operational system) timezone to UTC. 

If we want to have a coherent time variable in charts and logs and across computers with different time zones, one option would be using Universal Time when we parse the date string:

and create another object with the Chicago Time:

Sorry for the double post.
In the above post, I have attached an algorithm where we trade EURUSD based on whether London exchange or New York exchange are open. It is an example on how to use ConvertTo (a generalization of ConvertFromUtc) method to make trading decisions.

Hello Alexandre, 

Thanks a lot. 

Based on your explanation, I tried to trace the chart ploting code and find something strange.

The time parameter of the ChartPoint helper seems always being fed with datetime with kind always "unspecified".   

In BaseData Reader method, even I parse the time , convert it to utc and specifiying the kind as 'utc', when this ChartPoint helper is called, the kind is always "unspecified". 

Now since the kind is 'unspecified', the time conversion function inside ChartPoint helper assume the time given is local, even though it is utc indeed. Then it lookups OS timezone, add/subtract the OS timezone and return it to the caller.

I suspect this is a bug, but again, I am so new to both LEAN and C#, there are a lot of create that I may get it wrong. But I think the behavior should be something other than what we have now. 

https://github.com/QuantConnect/Lean/blob/master/Common/Charting.cs#L371

P.S. : Interestingly, when the ChartPoint caller is invoked with Equity plot data, the time parameter passed has kind set as 'utc'.  Not sure if this is related. 

Thanks

Hello Alexandre, 

Just in case, if the call stack when the problem happens maybe useful for you. I am attaching it here for your reference. 

Could you please try to set up the algorithm and the data subscription timezone?

I tested the attached algorithm with two different timezones in my PC. All results match: first data point of the day at 13:50 UTC.

Equity curve is plotted using a DateTime object that is specifically set to be UTC.
On the other hand, when we plot through the API it used a DateTime object that is not UTC unless we use the SetTimeZone method to set it.

Hello Alexandre,

Yes, in fact if you SetTimeZone(TimeZones.Utc), you will get the chart starts at 13:50UTC for any OS Timezone setting,  The TimeZones.Utc parameter for AddData seems does not affect the outcome.

While the SetTimeZone to utc gives the expected result, any other settings to SetTimeZone will unfortuanately almost guarantee to give the chart starting time un-decrytable. 

I experimented combinations of OS timezone and SetTimeZone have have the following result with some remarks for your team consideration to see if it can be rectified. 

Note: 

1 - 4: OS in NewYork timezone

5 - 8: OS in Shanghai timezone

9 - 11: for exploration purpose

utc - 4 : NewYork timezone in summer time

utc +8 : Shanghai

Observations:

A. We know that the "SPY" data start at

   9:30am (utc - 4) == 13:30 utc time == 21:30 (utc + 8)

B. Chart starts from 13:30 (i.e. showing utc time) when one of the two conditions is satisfied.

1. SetTimeZone = utc
2. OS Timezone = SetTimeZone (but can't interpret why a utc time is given with this combination)

C. Issues:

• I can't figure out what timezone the chart is showing in some of the combination above.  For example #5.  my OS Timezone is utc + 8 and I do not define SetTimeZone in my code.  The chart starts from 1:30am I have no idea how to interpret what it is. (all the time with question marks)
• Not clear of what timezone the chart is showing, maybe a legend may help.
• user need to combine multiple pieces of info from code, system time zone to determine what timezone the chart is showing. 
• when an algorithm is shared across multiple timezone, different users will very likely to have different timing display due to different OS timezone settings.
• Requirement of changing user Timezone to view the chart, in my opinion,  is very undesirable, it affect the rest of OS, applications, file saving record, etc running on the same platform.
• While UTC is good for processing and synchronization of multiple streams of datetime data from multiple timezone, it very unintuitive to read and analyse. When multiple time zone data are being processed, usually user will pick one of the local timezones as base timezone to 'display' the data. For example,  I operate in utc+8, import data from some sources in utc-5 and the exchange traded is in utc-4. I will usually covert all of them to UTC for internal processing first and the convert the result into the base timezone. I have never plotted my chart in utc timezone for display purpose.
• not able to display data in local time. ( I still cannot figure out the right combination to do that yet)
• many timezone settings inside LEAN, SetTimeZone, AddData timezone, SecurityExchange.timezone, etc.  How they interact and affect the output is not clear.

Thank you for you detailed reaseach.

When the OS time zone is UTC+8, the algorithm will think that it is 9:30 in Shangai timezone, therefore when it is translated to UTC, the result is 1:30 am.
When the algorithm time zone is UTC+8, the algorithm will thanslate the 9:30 in NY to 21:30 in NY (9:30 in Shangai) and, finally, when it is translated to UTC, it will fall atat 1:30 am too.

We can put your feature request on our TODO list.
Could you please create an issue in QuantConnect/Lean github page?

Hello Alexandre, 

Many thanks for your elaboration and patience. 

I have a little bit different view on that. 

Take the default "SPY" security data (2013/10/7) comes standard with LEAN download.

We know "SPY" started at 9:30am New York time that day (and every trading day indeed). Furthermore, assume the user operates at Shanghai/UTC+8 timezone, so the timezone setting of the host OS where LEAN is running is UTC+8.

We could express 9:30am NY time interchangeably in other timezones and they all mean the same instant of time. 

9:30am NY = 13:30 UTC = 21:30 Shanghai/UTC+8

So if we put that "SPY" data into LEAN and plot a chart with one of the three timezones we specified above, the chart should either starts at 9:30am,  13:30 or 21:30 depends on which one of the three timezones is specified in the algorithm.  Chart starts at any other time/timezone combinations would considered as incorrect. 

Now 1:30 of either (NY, UTC or Shanghai) is definitely not the time when the first tick of "SPY" happens according to the "equation" above.  

On the other hand, in order to make sense of 1:30 as a chart starting time, the timezone must be either UTC-12 or UTC+12. But none of these two timezone have we specified in our algorithm 

9:30am NY = 13:30 UTC = 21:30 Shanghai/UTC+8 = 1:30 UTC-12 = 1:30 UTC+12(+1day)

Therefore, I still feel there some miscalculation inside LEAN

Yes, I could create a ticket for this case. But please assist and review the ticket once I have created it. As I am not a native English speaker, I am afraid that I could not convey myself clear and describe the situation precisely. Will appreciate if you could help me a bit. 

Hello Alexandre Catarino,

I have created a ticket with link below. 

https://github.com/QuantConnect/Lean/issues/618

Many thanks and have a good weekend

HI Kow, Timezones are tricky but I think it may be a simple misunderstanding? Its not "130" its "1330" -- i.e. UTC +4, which during 6 months of the year equates to NYC time.

In LEAN we have:
 - Data Time Zone - What the data files are stored in
 - Algorithm Time Zone - What the algorithm is working in (Time property)
 - Exchange Time Zone - What the exchange hours are done in.

However the charting is always done in UTC. This is because JS timezones aren't rendered properly; so you must always store data for charting in UTC and then convert it as required. The browser doesn't maintain any reference to timezone so we do the conversion in the JS to Eastern time.

Because its always UTC the conversion from +8, and adding +4 to get to NYC is going to end up a mess. Setting the data timezone when you register it will shift it first, then shift it again for display.

99% of the users are in the US market; so we convert it to US hours which works for 99% of people. However we'll consider making an option to convert the local chartsfrom UTC to the Algorithm Time Zone which should address your needs.

Jared, txs for the explanation. I'd like to suggest adding a Display TZ which might default to Algorithm TZ, but the Algorithm might be being run on a server anywhere. By having the Display TZ all bases should be covered.

Hello Jared, 

Thanks for your comment. Let me show you how I observed "1:30" (not 13:30) as chart starting time.

I used the following code:

Note that AddSecurity method read and store "SPY" by itself, there's no way to change how the data is stored. 

So, there are really two time zone related factors that I can play with

a) SetTimeZone

b) host OS timezone

and here is the result:

Chart of #4, #5 and #6 start at 01:30.  

So if charting is displayed in UTC timezone, then all charts should start at 13:30UTC regardless tto how OS Timezone and SetTimeZone is set , right ?

We've pushed a PR here: https://github.com/QuantConnect/Lean/pull/751 which forces all data to be stored in UTC. We then force display of the charts in UTC-5 (EST). This way if you do your own custom plotting at least the source data will all be in UTC. This new code is now on master.

We've also updated the QuantConnect plots so custom plots are set UTC-5 hours - as 90% of the userbase are trading US markets.