Share New Research 
Hi. I want to ask you some help because in the last 2 weeks I am having daily crashes in a live trading strategy using Lean-Cli and IB. It always happens at 5.00 AM UTC. Below I leave what I consider the most important logs, and also the full log when the issue happen.
The same strategy is running on QuantConnect, and no error is being raised, only with Lean-cli. I have the latest lean (1.0.63) and on last Sunday I ran `lean live ` with `--update` param, but the issue persist.
Thank you for your help.
Main logs:
2021-09-01T05:01:00.2485757Z TRACE:: InteractiveBrokersBrokerage.HandleError(): RequestId: 3 ErrorCode: 511 - Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )
2021-09-01T05:01:00.2580050Z ERROR:: Brokerage.OnMessage(): Error - Code: 511 - Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )
2021-09-01T05:01:00.3144829Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exception: Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )Full Log:
2021-09-01T04:59:00.4910123Z TRACE:: Isolator.ExecuteWithTimeLimit(): Used: 62, Sample: 69, App: 424, CurrentTimeStepElapsed: 00:00.000
2021-09-01T05:00:00.0109848Z TRACE:: InteractiveBrokersBrokerage.OnIbAutomaterOutputDataReceived(): InteractiveBrokersBrokerage.IsWithinScheduledServerResetTimes(): False
2021-09-01T05:00:00.0210692Z TRACE:: InteractiveBrokersBrokerage.LookupSymbols(): Requesting symbol list for ES ...
2021-09-01T05:00:00.5241833Z TRACE:: Isolator.ExecuteWithTimeLimit(): Used: 60, Sample: 50, App: 423, CurrentTimeStepElapsed: 00:00.000
2021-09-01T05:01:00.0457944Z ERROR:: InteractiveBrokersBrokerage.FindContracts(): failed to receive response from IB within 60 seconds
2021-09-01T05:01:00.0511977Z TRACE:: InteractiveBrokersBrokerage.LookupSymbols(): Returning 0 contract(s) for ES
2021-09-01T05:01:00.1415626Z TRACE:: DataManager.RemoveSubscription(): Removed ES18H22,ES18H22,Minute,QuoteBar,Quote,Raw
2021-09-01T05:01:00.1488000Z TRACE:: DataManager.RemoveSubscription(): Removed ES18H22,ES18H22,Minute,TradeBar,Trade,Raw
2021-09-01T05:01:00.1574791Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): ES18H22
2021-09-01T05:01:00.1745717Z TRACE:: InteractiveBrokersBrokerage.Unsubscribe(): Unsubscribe Request: ES18H22
2021-09-01T05:01:00.2485757Z TRACE:: InteractiveBrokersBrokerage.HandleError(): RequestId: 3 ErrorCode: 511 - Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )
2021-09-01T05:01:00.2580050Z ERROR:: Brokerage.OnMessage(): Error - Code: 511 - Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )
2021-09-01T05:01:00.3144829Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exception: Cancel Market Data Sending Error - . Origin: [Id=3] Subscribe: ES18H22 (FUT ES USD GLOBEX 20220318 0 )
2021-09-01T05:01:00.5465038Z TRACE:: Isolator.ExecuteWithTimeLimit(): Used: 61, Sample: 68, App: 436, CurrentTimeStepElapsed: 01:00.024
2021-09-01T05:01:01.4005097Z TRACE:: InteractiveBrokersBrokerage.Connect(): IB message processing thread ended: #16
2021-09-01T05:01:01.4027963Z TRACE:: InteractiveBrokersBrokerage.HandleConnectionClosed(): API client disconnected [Server Version: 0].
2021-09-01T05:01:01.4255197Z TRACE:: DataManager.RemoveSubscription(): Removed ES18H22,ES18H22,Minute,OpenInterest,OpenInterest,Raw,Internal
2021-09-01T05:01:01.4294680Z TRACE:: DataManager.RemoveSubscription(): Removed ES17Z21,ES17Z21,Minute,QuoteBar,Quote,Raw
2021-09-01T05:01:01.4310787Z TRACE:: DataManager.RemoveSubscription(): Removed ES17Z21,ES17Z21,Minute,TradeBar,Trade,Raw
2021-09-01T05:01:01.4332062Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): ES17Z21
2021-09-01T05:01:01.4339331Z TRACE:: InteractiveBrokersBrokerage.Unsubscribe(): Unsubscribe Request: ES17Z21
2021-09-01T05:01:01.4347799Z TRACE:: InteractiveBrokersBrokerage.HandleError(): RequestId: -1 ErrorCode: 504 - Not connected
2021-09-01T05:01:01.4368118Z ERROR:: Brokerage.OnMessage(): Error - Code: 504 - Not connected
2021-09-01T05:01:01.4406226Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exc
eption: Not connected
2021-09-01T05:01:01.4427968Z TRACE:: DataManager.RemoveSubscription(): Removed ES17Z21,ES17Z21,Minute,OpenInterest,OpenInterest,Raw,Internal
2021-09-01T05:01:01.4523611Z ERROR:: AlgorithmManager.Run(): Stopping, encountered a runtime error at 9/1/2021 5:00:00 AM UTC.
2021-09-01T05:01:01.4546413Z TRACE:: Engine.Run(): Exiting Algorithm Manager
2021-09-01T05:01:01.4616049Z TRACE:: LiveTradingDataFeed.Exit(): Start. Setting cancellation token...
2021-09-01T05:01:01.4642240Z TRACE:: BaseDataExchange(CustomDataExchange) Stopping...
2021-09-01T05:01:01.4653745Z TRACE:: LiveTradingDataFeed.Exit(): Exit Finished.
2021-09-01T05:01:01.4659487Z TRACE:: Engine.Run(): Sending runtime error to user...
2021-09-01T05:01:01.4730533Z TRACE:: BaseDataExchange(CustomDataExchange).ConsumeQueue(): Cancellation requested. Exiting...
2021-09-01T05:01:01.4789479Z TRACE:: StopSafely(): waiting for 'Transaction Thread' thread to stop...
2021-09-01T05:01:01.4844481Z TRACE:: BrokerageTransactionHandler.Run(): Ending Thread...
2021-09-01T05:01:01.4879553Z TRACE:: StopSafely(): waiting for 'RealTime Thread' thread to stop...
2021-09-01T05:01:01.4956047Z TRACE:: Log: Runtime Error: Not connected
2021-09-01T05:01:01.4961247Z ERROR:: Runtime Error: Not connected
Not connected
2021-09-01T05:01:02.0027769Z TRACE:: LiveTradingRealTimeHandler.Run(): Exiting thread... Exit triggered: True
2021-09-01T05:01:02.0123264Z TRACE:: DataManager.RemoveSubscription(): Removed QC-UNIVERSE-USERDEFINED-USA-BASE,QC-UNIVERSE-USERDEFINED-USA-BASE,Daily,CBOE,Trade,Adjusted,Internal
2021-09-01T05:01:02.0210112Z TRACE:: DataManager.RemoveSubscription(): Removed VIX,VIX,Daily,CBOE,Trade,Adjusted
2021-09-01T05:01:02.0313251Z TRACE:: DataManager.RemoveSubscription(): Removed QC-UNIVERSE-USERDEFINED-USA-EQUITY,QC-UNIVERSE-USERDEFINED-USA-EQUITY,Minute,TradeBar,Trade,Adjusted,Internal
2021-09-01T05:01:02.0399182Z TRACE:: DataManager.RemoveSubscription(): Removed SPY,SPY,Daily,TradeBar,Trade,Adjusted
2021-09-01T05:01:02.0405906Z TRACE:: DataManager.RemoveSubscription(): Removed SPY,SPY,Minute,TradeBar,Trade,Adjusted,Internal
2021-09-01T05:01:02.0447006Z TRACE:: DataManager.RemoveSubscription(): Removed SVXY,SVXY,Minute,TradeBar,Trade,Raw
2021-09-01T05:01:02.0459142Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): SVXY
2021-09-01T05:01:02.0462761Z TRACE:: InteractiveBrokersBrokerage.Unsubscribe(): Unsubscribe Request: SVXY
2021-09-01T05:01:02.0464703Z TRACE:: InteractiveBrokersBrokerage.HandleError(): RequestId: -1 ErrorCode: 504 - Not connected
2021-09-01T05:01:02.0480952Z ERROR:: Brokerage.OnMessage(): Error - Code: 504 - Not connected
2021-09-01T05:01:02.0544492Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exception: Not connected
2021-09-01T05:01:02.0544492Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exception: Not connected
2021-09-01T05:01:02.0548779Z TRACE:: DataManager.RemoveSubscription(): Removed SVXY,SVXY,Minute,QuoteBar,Quote,Raw
2021-09-01T05:01:02.0563421Z TRACE:: DataManager.RemoveSubscription(): Removed ES17U21,ES17U21,Minute,TradeBar,Trade,Raw
2021-09-01T05:01:02.0599116Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): /ES
2021-09-01T05:01:02.0623245Z TRACE:: InteractiveBrokersBrokerage.Unsubscribe(): Unsubscribe Request: /ES
2021-09-01T05:01:02.0635285Z TRACE:: DataManager.RemoveSubscription(): Removed /ES,/ES,Minute,ZipEntryName,Quote,Raw
2021-09-01T05:01:02.0637850Z TRACE:: DataManager.RemoveSubscription(): Removed /QC-UNIVERSE-USERDEFINED-CME-FUTURE,/QC-UNIVERSE-USERDEFINED-CME-FUTURE,Minute,QuoteBar,Quote,Raw,Internal
2021-09-01T05:01:02.0643169Z TRACE:: DataManager.RemoveSubscription(): Removed ES17U21,ES17U21,Minute,QuoteBar,Quote,Raw
2021-09-01T05:01:02.0661696Z TRACE:: EventBasedDataQueueHandlerSubscriptionManager.Unsubscribe(): ES17U21
2021-09-01T05:01:02.0664922Z TRACE:: InteractiveBrokersBrokerage.Unsubscribe(): Unsubscribe Request: ES17U21
2021-09-01T05:01:02.0668759Z TRACE:: InteractiveBrokersBrokerage.HandleError(): RequestId: -1 ErrorCode: 504 - Not connected
2021-09-01T05:01:02.0679109Z ERROR:: Brokerage.OnMessage(): Error - Code: 504 - Not connected
2021-09-01T05:01:02.0734619Z ERROR:: Extensions.SetRuntimeError(): Extensions.SetRuntimeError(): RuntimeError at 9/1/2021 5:00:00 AM UTC. Context: Brokerage Error System.Exception: Not connected
2021-09-01T05:01:02.0771883Z TRACE:: DataManager.RemoveSubscription(): Removed ES17U21,ES17U21,Minute,OpenInterest,OpenInterest,Raw,Internal
2021-09-01T05:01:02.0905236Z TRACE:: DefaultAlphaHandler.Exit(): Exiting...
2021-09-01T05:01:02.0945000Z TRACE:: DefaultAlphaHandler.Exit(): Ended
2021-09-01T05:01:02.1006179Z TRACE:: LiveTradingResultHandler.Run(): Ending Thread...
2021-09-01T05:01:02.1041788Z TRACE:: StopSafely(): waiting for 'Result Thread' thread to stop...
2021-09-01T05:01:02.1152049Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Starting...
2021-09-01T05:01:02.6918964Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Finished storing results. Start sending...
2021-09-01T05:01:02.6947847Z TRACE:: LiveTradingResultHandler.SendFinalResult(): Ended
2021-09-01T05:01:02.6979844Z TRACE:: Engine.Run(): Disconnecting from brokerage...
2021-09-01T05:01:02.6998665Z TRACE:: InteractiveBrokersBrokerage.HandleConnectionClosed(): API client disconnected [Server Version: 0].
2021-09-01T05:01:02.7032352Z TRACE:: InteractiveBrokersBrokerage.Dispose(): Disposing of IB resources.
2021-09-01T05:01:02.7039267Z TRACE:: InteractiveBrokersBrokerage.HandleConnectionClosed(): API client disconnected [Server Version: 0].
2021-09-01T05:01:02.7571646Z TRACE:: InteractiveBrokersBrokerage.OnIbAutomaterOutputDataReceived(): Updating IBGateway configuration file: /usr/local/ibgateway/ibgateway.vmoptions
2021-09-01T05:01:02.7722916Z TRACE:: Engine.Run(): Disposing of setup handler...
2021-09-01T05:01:02.7819056Z TRACE:: BrokerageSetupHandler.Setup(): Found data queue handler to dispose: QuantConnect.Brokerages.InteractiveBrokers.InteractiveBrokersBrokerage
2021-09-01T05:01:02.7870703Z TRACE:: Engine.Main(): Analysis Completed and Results Posted.
2021-09-01T05:01:02.7914385Z TRACE:: StopSafely(): waiting for 'Transaction Thread' thread to stop...
2021-09-01T05:01:02.8122851Z TRACE:: Engine.Main(): Packet removed from queue: L-3985613767
2021-09-01T05:01:02.8200710Z TRACE:: LeanEngineSystemHandlers.Dispose(): start...
2021-09-01T05:01:02.8296350Z TRACE:: LeanEngineSystemHandlers.Dispose(): Disposed of system handlers.
2021-09-01T05:01:02.8362422Z TRACE:: LeanEngineAlgorithmHandlers.Dispose(): start...
2021-09-01T05:01:02.8420795Z TRACE:: BrokerageSetupHandler.Setup(): Found data queue handler to dispose: QuantConnect.Brokerages.InteractiveBrokers.InteractiveBrokersBrokerage
2021-09-01T05:01:02.8663116Z TRACE:: LeanEngineAlgorithmHandlers.Dispose(): Disposed of algorithm handlers.
Varad Kabade
Hi Gabriel Moncarz,
For issues with live trading, please contact support@quantconnect.com with the live deployment attached.
Best,
Varad Kabade
Gabriel Moncarz
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!