This incident was caused by resource contention in our orchestration layer. One of the components had a memory leak which resulted in a host using large amounts of swap which put additional stress on the other services run on the host. As a result, some hosts rebooted as they lost their connection to the orchestration layer. This incident affected 16% of the EU-2 region hosts.

Fixes deployed during maintenance on Wednesday November 21st:

• Split orchestration components on dedicated hosts
• Fix for the memory leak in the coordinator process.
• Fix for a bug in an upstream library used in the orchestration software.
• Fix for a networking bug preventing project recovery after reallocation.

‌

We are continuing to monitor the situation closely to ensure the issue does not present again.