When running live trading with GDAX from stand-alone LEAN instances, I frequently notice websocket disconnection (example logs below). The situation appears to be quite bad in us-east zones (Google Cloud instances) with disconnects happening every few minutes. Zones in us-west seem a bit better but still disconnecting every hour or so.

What are the recommended practices for running LEAN instances with GDAX in terms of geo-location / cloud zones? I found some discussions mentioning websocket provision with cloudflare (supposed to work in multiple regious right?) but no specific conclusions on achieving the websocket stability.

Example log fragments:

20180609 02:18:58.877 Trace:: 2018-06-09 02:18:58Z Isolator.ExecuteWithTimeLimit(): Used: 47 Sample: 43 20180609 02:19:59.049 Trace:: 2018-06-09 02:19:59Z Isolator.ExecuteWithTimeLimit(): Used: 47 Sample: 46 20180609 02:20:54.009 Trace:: Debug: 6/9/2018 2:20:53 AM|Fatal|WebSocket.<startReceiving>b__170_2|System.NullReferenceException: Object reference not set to an inst ance of an object at WebSocketSharp.WebSocket.<startReceiving>b__170_2 (System.Exception ex) [0x00001] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.Ext.ReadBytesAsync (System.IO.Stream stream, System.Int32 length, System.Action`1[T] completed, System.Action`1[T] err or) [0x00093] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readPayloadDataAsync (System.IO.Stream stream, WebSocketSharp.WebSocketFrame frame, System.Action`1[T] completed, System.Action`1[T] error) [0x00097] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame+<>c__DisplayClass82_0.<ReadFrameAsync>b__2 (WebSocketSharp.WebSocketFrame frame2) [0x00000] in <1fc1e12 f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readMaskingKeyAsync (System.IO.Stream stream, WebSocketSharp.WebSocketFrame frame, System.Action`1[T] c ompleted, System.Action`1[T] error) [0x0004a] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame+<>c__DisplayClass82_0.<ReadFrameAsync>b__1 (WebSocketSharp.WebSocketFrame frame1) [0x00000] in <1fc1e12 f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame.readExtendedPayloadLengthAsync (System.IO.Stream stream, WebSocketSharp.WebSocketFrame frame, System.Ac tion`1[T] completed, System.Action`1[T] error) [0x00044] in <1fc1e12f21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame+<>c__DisplayClass82_0.<ReadFrameAsync>b__0 (WebSocketSharp.WebSocketFrame frame) [0x00000] in <1fc1e12f 21eb4681aa69c484f58d0121>:0 at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x0000c] in <1fc1e12f21eb4681aa69c48 4f58d0121>:0 at WebSocketSharp.Ext+<>c__DisplayClass54_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x0009e] in <1fc1e12f21eb4681aa69c484f58d0121 >:0 20180609 02:20:59.226 Trace:: 2018-06-09 02:20:59Z Isolator.ExecuteWithTimeLimit(): Used: 47 Sample: 40 20180609 02:21:59.281 Trace:: 2018-06-09 02:21:59Z Isolator.ExecuteWithTimeLimit(): Used: 47 Sample: 52 20180609 02:22:24.212 Trace:: Brokerage.OnMessage(): Disconnect - Code: Disconnect - Connection with server lost. This could be because of internet connectivity iss ues. 20180609 02:22:24.212 Trace:: DefaultBrokerageMessageHandler.Handle(): Disconnected. 20180609 02:22:24.212 Trace:: DefaultBrokerageMessageHandler.Handle(): Disconnect when exchanges are open, trying to reconnect for 15 minutes. 20180609 02:22:25.725 Trace:: BaseWebsocketsBrokerage(): Reconnecting... IsConnected: False 20180609 02:22:26.432 Trace:: WebSocketWrapper.OnOpen(): Connection opened(True): wss://ws-feed.gdax.com 20180609 02:22:26.433 Trace:: Brokerage.OnMessage(): Information - Code: -1 - GDAXBrokerage.Subscribe: Sent subscribe. 20180609 02:22:27.018 Trace:: Debug: Brokerage Info: GDAXBrokerage.Subscribe: Sent subscribe. 20180609 02:22:27.311 Trace:: Brokerage.OnMessage(): Information - Code: -1 - GDAXBrokerage.Subscribe: Sent subscribe. 20180609 02:22:28.030 Trace:: Debug: Brokerage Info: GDAXBrokerage.Subscribe: Sent subscribe. 20180609 02:22:34.290 Trace:: Brokerage.OnMessage(): Reconnect - Code: Reconnect - Connection with server restored. 20180609 02:22:34.291 Trace:: DefaultBrokerageMessageHandler.Handle(): Reconnected. 20180609 02:22:59.427 Trace:: 2018-06-09 02:22:59Z Isolator.ExecuteWithTimeLimit(): Used: 48 Sample: 47 20180609 02:23:59.554 Trace:: 2018-06-09 02:23:59Z Isolator.ExecuteWithTimeLimit(): Used: 48 Sample: 41

Author