At 2am last night there was an issue with Google Cloud DB backup which caused our API to go down for 20 minutes. This caused a segment of the live running algorithms to trigger a timeout error and terminate to be safe. All affected users were immediately emailed and the majority of you have redeployed your algorithm by US Market Open this morning.

Although it can be annoying the code is setup to err on the side of caution and terminate in cases like this to prevent your algorithm being out of contact. Our API has had great uptime generally and we're at all time highs in all our key metrics.

We will always be aiming for 100% uptime; but in reality even with 99.9999% performance there will be downtime eventually and algorithms should be designed to be stateless -- i.e. easily re-deployed and initialized in cases like this. We're working with Google Engineers to figure out the cause of the DB outage and prevent it happening in the future.

Thank you for your patience as we continue forging the frontiers of quant trading!

Jared & Team @ QuantConnect

Author