On January 16th we released an upgrade to our orchestration layer which added new important capabilities. These improvements allow us to handle higher deployment volumes more efficiently, scaling our platform to a new level. Unfortunately, this release contained a regression.
Between January 18th and January 24th, customers across regions CA-1, US-2, AU, DE-1 and EU-2 reported issues when reactivating development environments that were previously inactive or in some cases populating database relationships. Other customers also reported problems when sending email. We identified and mitigated these issues as they occurred while working on the definitive bug fixes that were deployed in all affected regions on January 28th, restoring all functionality back to normal.
The impact experienced by most of our customers was low, though a couple customers had a longer downtime due to misdiagnosed issues. Mislead by the timing, we incorrectly diagnosed the downtime to be related to the maintenance when it wasn’t and advised them incorrectly. We apologize to these customers. We highly encourage all of our customers to open an urgent ticket if their environment is down for more than 20 minutes, even within a maintenance window, as any downtime taking longer than the build and deploy hook times needs to be investigated by our Support. Investigation into sites with Premium support SLA will be conducted automatically based on monitoring alerts.