Discussion Forum - QuantConnect.com

Kow Gate INVESTOR

| Reward Discussion

Hello experts,

I'm new to QC so please bear with me if I ask stupid questions.

I try to import custom data and plot the data in QC. I tried both the online QC Algorithm Lab and with LEAN engine download in my own desktop. The plot give different results.

Below is the test code I use. When run the backtest & plot from here online, the plot start from 2016/07/16 8:50am which is what I am expecting,

But when using the same code and plot using LEAN engine in my desktop, the plot start from 2016/0716 00:50am

Is the difference of behavior expected? What should I do to obtain consistent plot result ?

Many thanks

Note: In my attached backtest, I can't figure how to show the plot I am referring to. However, it you clone the backtest and then : Select Chart -> Market Price Plot, you will see the plot of custom data I am referring to.

I also attach a screen captures here. First one is result of plot on cloud and the second one is the plot in using LEAN desktop.

Thanks

Update Backtest

Kow Gate

Accepted Answer

Update Backtest

Notebook

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Wink Saville

267 ,

I'm guessing, but I'd say it's local time vs UTC.

Wink Saville

267 ,

I found this link searching the inet for setting timezone on quantconnect. I to am a noob, so I'm very likely wrong :)

Kow Gate

368 ,

Hello Wink,

Thanks for your comment.

The problem is I am using the *same* piece of code in both platforms to do the test. I expect I would get the same result from both tests no matter the results are right or wrong.

But in this case, the result does not agree with each other. That's what I am trying to understand. :)

Wink Saville

267 ,

Yea, I agree it would nice if they were the same and the problem may have nothing to do with my timezone guess. We'll have to see what the experts say.

Alexandre Catarino

100.3k ,

Lean has an extension methods that can be used to make timezone convertions.
In this case, you could use ConvertFromUtc:

public override BaseData Reader(SubscriptionDataConfig config, 

    string line, System.DateTime date, bool isLive)

{

    /* ... */



    rcBar.Time = DateTime.ParseExact(data[0], 

        "yyyy-MM-dd HH:mm:sszz", 

        CultureInfo.InvariantCulture)

        // Convert to Chicago Timezone from UTC

        .ConvertFromUtc(TimeZones.Chicao);



    /* ... */

}

This solution has fixed the mismatch issue for me.

Kow Gate

368 ,

Hello Alexandre,

Thanks a lot for your guidance.

First of all, I made some minor changes in your code, see comment below about the changes.

/*

* Imported custom data is in local time with UTC offset format, 

* some minor change made here to turn the parse output into UTC,

* before feeding to ConvertFromUtc()

* 

* rcBar.Time = DateTime.ParseExact(data[0], "yyyy-MM-dd HH:mm:sszz", 

*                               CultureInfo.InvariantCulture)

*                               .ConvertFromUtc(TimeZones.Chicao);

*/



rcBar.Time = DateTime.ParseExact(data[0], "yyyy-MM-dd HH:mm:sszz",

                                CultureInfo.InvariantCulture,

                                DateTimeStyles.AdjustToUniversal)

                                .ConvertFromUtc(TimeZones.Chicao);

Now with that change, the 'cloud' version of LEAN plot output agrees with the log , they both start at 8:50am 07/26/2016.

Now the problem again is on LEAN desktop version. Timing of the plot does not agrees with the log. In this case, I have the plot starts at 00:50am 07/26/2016 while the log prints 08:50am.

I tried to convet the UTC to other timezone such as NewYork, the disagreement still there. Plot start at 01:50am while the log prints start at 09:50am.

One more interesting thing I find out is that, the plot of LEAN desktop seems to change according to the time zone of setting of my PC.

For example, when my PC timezone is set to UTC+08, plot starts at 00:50am while log starts at 08:50am.

When I change my PC timezone to UTC-5, plot starts at 12:50pm while log starts at 08:50am.

The log seems to give the result I want but the plot seems never have it done right.

One more thing I want to point out , not sure if that relates to this problem, There's no sample of import data until 20 minutes after market open. (i.e. the first data is received at 09:50am Eastern Time. Just want to highlight this in case it is related.

Why this inconsistency happens? Are there any additional setting we need to invoke when we move the code from cloud to desktop?

Many thanks

Kow Gate

368 ,

anyone has idea on this? I digged into the Community but seems nobody has problem with this.

Not sure if most of the users are using daily data (and hence no need to handle the timezone issue) or I have overlooked/missed some config/settings here.

Many thanks eveybody

Alexandre Catarino

100.3k ,

When we convert a string to a DateTime object, using DateTime.ParseExact, the result will take into consideration the operational system timezone (link) and DateTimeKind will be set to Local or Unspecified, unless we specify the style, like you did, using DateTimeStyles.AdjustToUniversal.

However, when we use the ConvertFromUtc method to convert to Chicago Time from UTC, DateTimeKind is set to Unspecified.

Why does the DateTimeKind matter? Because when we plot the data, each data point time coordinate will be converted to UTC by the DateTime.ToUniversalTime method here. If DateTimeKind is not Utc, this method will assume we want to convert local (operational system) timezone to UTC.

If we want to have a coherent time variable in charts and logs and across computers with different time zones, one option would be using Universal Time when we parse the date string:

rcBar.Time = DateTime.ParseExact(data[0], 

    "yyyy-MM-dd HH:mm:sszz",

    CultureInfo.InvariantCulture,

    DateTimeStyles.AdjustToUniversal);

and create another object with the Chicago Time:

Log(string.Format("{0} -> {1}: ${2}", 

    data.Time.ConvertFromUtc(TimeZones.Chicao), 

    data.Symbol, 

    data.Value));

Alexandre Catarino

100.3k ,

Sorry for the double post.
In the above post, I have attached an algorithm where we trade EURUSD based on whether London exchange or New York exchange are open. It is an example on how to use ConvertTo (a generalization of ConvertFromUtc) method to make trading decisions.

Kow Gate

368 ,

Hello Alexandre,

Thanks a lot.

Based on your explanation, I tried to trace the chart ploting code and find something strange.

The time parameter of the ChartPoint helper seems always being fed with datetime with kind always "unspecified".

In BaseData Reader method, even I parse the time , convert it to utc and specifiying the kind as 'utc', when this ChartPoint helper is called, the kind is always "unspecified".

Now since the kind is 'unspecified', the time conversion function inside ChartPoint helper assume the time given is local, even though it is utc indeed. Then it lookups OS timezone, add/subtract the OS timezone and return it to the caller.

I suspect this is a bug, but again, I am so new to both LEAN and C#, there are a lot of create that I may get it wrong. But I think the behavior should be something other than what we have now.

https://github.com/QuantConnect/Lean/blob/master/Common/Charting.cs#L371

///Constructor for datetime-value arguements:

public ChartPoint(DateTime time, decimal value) 

{

   x = Convert.

     ToInt64(Time.DateTimeToUnixTimeStamp(time.ToUniversalTime()));

   y = value.SmartRounding();

}

P.S. : Interestingly, when the ChartPoint caller is invoked with Equity plot data, the time parameter passed has kind set as 'utc'. Not sure if this is related.

Thanks

Kow Gate

368 ,

Hello Alexandre,

Just in case, if the call stack when the problem happens maybe useful for you. I am attaching it here for your reference.

>	QuantConnect.Common.dll!QuantConnect.ChartPoint.ChartPoint(System.DateTime time, decimal value) Line 374	C#

 	QuantConnect.Common.dll!QuantConnect.Series.AddPoint(System.DateTime time, decimal value, bool liveMode) Line 291	C#

 	QuantConnect.Algorithm.dll!QuantConnect.Algorithm.QCAlgorithm.Plot(string chart, string series, decimal value) Line 190	C#

 	QuantConnect.Algorithm.CSharp.dll!QuantConnect.Algorithm.CSharp.DataTimingProblemAlex.OnData(QuantConnect.Algorithm.CSharp.DataTimingProblemAlex.RCL data) Line 55	C#

 	[External Code]	

 	QuantConnect.Lean.Engine.dll!QuantConnect.Lean.Engine.AlgorithmManager.Run(QuantConnect.Packets.AlgorithmNodePacket job, QuantConnect.Interfaces.IAlgorithm algorithm, QuantConnect.Lean.Engine.DataFeeds.IDataFeed feed, QuantConnect.Lean.Engine.TransactionHandlers.ITransactionHandler transactions, QuantConnect.Lean.Engine.Results.IResultHandler results, QuantConnect.Lean.Engine.RealTime.IRealTimeHandler realtime, QuantConnect.Interfaces.ICommandQueueHandler commands, System.Threading.CancellationToken token) Line 518	C#

 	QuantConnect.Lean.Engine.dll!QuantConnect.Lean.Engine.Engine.Run.AnonymousMethod__2() Line 225	C#

 	[External Code]

Alexandre Catarino

100.3k ,

Could you please try to set up the algorithm and the data subscription timezone?

# In Initialize method

SetTimeZone(TimeZones.Utc);

AddData<RCL>(_co, Resolution.Minute, TimeZones.Utc);

I tested the attached algorithm with two different timezones in my PC. All results match: first data point of the day at 13:50 UTC.

Equity curve is plotted using a DateTime object that is specifically set to be UTC.
On the other hand, when we plot through the API it used a DateTime object that is not UTC unless we use the SetTimeZone method to set it.

Kow Gate

368 ,

Hello Alexandre,

Yes, in fact if you SetTimeZone(TimeZones.Utc), you will get the chart starts at 13:50UTC for any OS Timezone setting, The TimeZones.Utc parameter for AddData seems does not affect the outcome.

While the SetTimeZone to utc gives the expected result, any other settings to SetTimeZone will unfortuanately almost guarantee to give the chart starting time un-decrytable.

I experimented combinations of OS timezone and SetTimeZone have have the following result with some remarks for your team consideration to see if it can be rectified.





   OS Timezone SetTimeZone First chart point Time

1  utc - 4     n/a         13:30

2  utc - 4     utc - 4     13:30

3  utc - 4     utc         13:30

4  utc - 4     utc + 8     1:30 (?)

5  utc + 8     n/a         1:30 (?)

6  utc + 8     utc - 4     1:30 (?)

7  utc + 8     utc         13:30

8  utc + 8     utc + 8     13:30

9  utc - 4     utc + 9     2:30 (?)

10 utc - 5     utc + 8     2:30 (?)

11 utc - 5     utc + 8     3:30 (?)

Note:

1 - 4: OS in NewYork timezone

5 - 8: OS in Shanghai timezone

9 - 11: for exploration purpose

utc - 4 : NewYork timezone in summer time

utc +8 : Shanghai

Observations:

A. We know that the "SPY" data start at

9:30am (utc - 4) == 13:30 utc time == 21:30 (utc + 8)

B. Chart starts from 13:30 (i.e. showing utc time) when one of the two conditions is satisfied.

SetTimeZone = utc
OS Timezone = SetTimeZone (but can't interpret why a utc time is given with this combination)

C. Issues:

I can't figure out what timezone the chart is showing in some of the combination above. For example #5. my OS Timezone is utc + 8 and I do not define SetTimeZone in my code. The chart starts from 1:30am I have no idea how to interpret what it is. (all the time with question marks)
Not clear of what timezone the chart is showing, maybe a legend may help.
user need to combine multiple pieces of info from code, system time zone to determine what timezone the chart is showing.
when an algorithm is shared across multiple timezone, different users will very likely to have different timing display due to different OS timezone settings.
Requirement of changing user Timezone to view the chart, in my opinion, is very undesirable, it affect the rest of OS, applications, file saving record, etc running on the same platform.
While UTC is good for processing and synchronization of multiple streams of datetime data from multiple timezone, it very unintuitive to read and analyse. When multiple time zone data are being processed, usually user will pick one of the local timezones as base timezone to 'display' the data. For example, I operate in utc+8, import data from some sources in utc-5 and the exchange traded is in utc-4. I will usually covert all of them to UTC for internal processing first and the convert the result into the base timezone. I have never plotted my chart in utc timezone for display purpose.
not able to display data in local time. ( I still cannot figure out the right combination to do that yet)
many timezone settings inside LEAN, SetTimeZone, AddData timezone, SecurityExchange.timezone, etc. How they interact and affect the output is not clear.

Alexandre Catarino

100.3k ,

Thank you for you detailed reaseach.

When the OS time zone is UTC+8, the algorithm will think that it is 9:30 in Shangai timezone, therefore when it is translated to UTC, the result is 1:30 am.
When the algorithm time zone is UTC+8, the algorithm will thanslate the 9:30 in NY to 21:30 in NY (9:30 in Shangai) and, finally, when it is translated to UTC, it will fall atat 1:30 am too.

We can put your feature request on our TODO list.
Could you please create an issue in QuantConnect/Lean github page?

Kow Gate

368 ,

Hello Alexandre,

Many thanks for your elaboration and patience.

I have a little bit different view on that.

Take the default "SPY" security data (2013/10/7) comes standard with LEAN download.

We know "SPY" started at 9:30am New York time that day (and every trading day indeed). Furthermore, assume the user operates at Shanghai/UTC+8 timezone, so the timezone setting of the host OS where LEAN is running is UTC+8.

We could express 9:30am NY time interchangeably in other timezones and they all mean the same instant of time.

9:30am NY = 13:30 UTC = 21:30 Shanghai/UTC+8

So if we put that "SPY" data into LEAN and plot a chart with one of the three timezones we specified above, the chart should either starts at 9:30am, 13:30 or 21:30 depends on which one of the three timezones is specified in the algorithm. Chart starts at any other time/timezone combinations would considered as incorrect.

Now 1:30 of either (NY, UTC or Shanghai) is definitely not the time when the first tick of "SPY" happens according to the "equation" above.

On the other hand, in order to make sense of 1:30 as a chart starting time, the timezone must be either UTC-12 or UTC+12. But none of these two timezone have we specified in our algorithm

9:30am NY = 13:30 UTC = 21:30 Shanghai/UTC+8 = 1:30 UTC-12 = 1:30 UTC+12(+1day)

Therefore, I still feel there some miscalculation inside LEAN

Yes, I could create a ticket for this case. But please assist and review the ticket once I have created it. As I am not a native English speaker, I am afraid that I could not convey myself clear and describe the situation precisely. Will appreciate if you could help me a bit.

Kow Gate

368 ,

Hello Alexandre Catarino,

I have created a ticket with link below.

https://github.com/QuantConnect/Lean/issues/618

Many thanks and have a good weekend

Jared Broad

STAFF ,

HI Kow, Timezones are tricky but I think it may be a simple misunderstanding? Its not "130" its "1330" -- i.e. UTC +4, which during 6 months of the year equates to NYC time.

In LEAN we have:
- Data Time Zone - What the data files are stored in
- Algorithm Time Zone - What the algorithm is working in (Time property)
- Exchange Time Zone - What the exchange hours are done in.

However the charting is always done in UTC. This is because JS timezones aren't rendered properly; so you must always store data for charting in UTC and then convert it as required. The browser doesn't maintain any reference to timezone so we do the conversion in the JS to Eastern time.

Because its always UTC the conversion from +8, and adding +4 to get to NYC is going to end up a mess. Setting the data timezone when you register it will shift it first, then shift it again for display.

99% of the users are in the US market; so we convert it to US hours which works for 99% of people. However we'll consider making an option to convert the local chartsfrom UTC to the Algorithm Time Zone which should address your needs.

Wink Saville

267 ,

Jared, txs for the explanation. I'd like to suggest adding a Display TZ which might default to Algorithm TZ, but the Algorithm might be being run on a server anywhere. By having the Display TZ all bases should be covered.

Kow Gate

368 ,

Hello Jared,

Thanks for your comment. Let me show you how I observed "1:30" (not 13:30) as chart starting time.

I used the following code:

namespace QuantConnect

{   

    public class DataTimingProblemDefaultSpy : QCAlgorithm

    {

        public override void Initialize() 

        {

            SetStartDate(2013, 10, 7);

            SetEndDate(2013, 10, 8);

            

            SetCash(25000);

            

            SetTimeZone(TimeZones.Shanghai);

            

            // "SPY" data comes standard with LEAN

            AddSecurity(SecurityType.Equity, "SPY", Resolution.Minute);

            

            var stockPlot = new Chart("Market Price Plot");

            var assetPrice = new Series("Price", SeriesType.Line, 0);

            var rcCA1 = new Series("C1");



            stockPlot.AddSeries(assetPrice);



            AddChart(stockPlot);

        }



        public void OnData(TradeBars data) 

        {   

            var spy = data["SPY"];

            Plot("Market Price Plot", "Price", spy.Price);

            Log(string.Format("{0} -> {1}: ${2} {3}", spy.Time, spy.Symbol, spy.Value, spy.Time.Kind));

        }

    }

}

Note that AddSecurity method read and store "SPY" by itself, there's no way to change how the data is stored.

So, there are really two time zone related factors that I can play with

a) SetTimeZone

b) host OS timezone

and here is the result:

  OS Timezone   SetTimeZone   First chart point Time

1 utc - 4       not specified 13:30

2 utc - 4       utc - 4       13:30

3 utc - 4       utc           13:30

4 utc - 4       utc + 8       1:30



5 utc + 8       not specified 1:30

6 utc + 8       utc - 4       1:30

7 utc + 8       utc           13:30

8 utc + 8       utc + 8       13:30

Chart of #4, #5 and #6 start at 01:30.

So if charting is displayed in UTC timezone, then all charts should start at 13:30UTC regardless tto how OS Timezone and SetTimeZone is set , right ?

Jared Broad

STAFF ,

We've pushed a PR here: https://github.com/QuantConnect/Lean/pull/751 which forces all data to be stored in UTC. We then force display of the charts in UTC-5 (EST). This way if you do your own custom plotting at least the source data will all be in UTC. This new code is now on master.

We've also updated the QuantConnect plots so custom plots are set UTC-5 hours - as 90% of the userbase are trading US markets.

Kow Gate INVESTOR

Update Backtest

Notebook

To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!

Platform

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240000 Quants.

Online Now

Stay Connected

Different behavior for custom data plot using QC in cloud and desktop

Started By:

Join QuantConnect for Free Sign Up

Platform

SIGN IN

QuantConnect Community

Alpha League Competition: $1,000 Weekly Prize Pool

Filter Discussion by Tags

240000 Quants.

Online Now

Stay Connected

Different behavior for custom data plot using QC in cloud and desktop

Started By:

Join QuantConnect for Free Sign Up