Partial Outage on
Incident Report for

What happened

We added object storage devices to the region. This change unexpectedly caused I/O performance issues.

Customer impact

Affected environments experienced slower than usual read and write operations, or their writable mounts suddenly became read-only.

Resolution steps

The response team added more object storage, allowing the data recalibration to complete faster, and to provide more I/O throughput to affected projects. Read-only mounts were manually investigated and recovered.

Posted Nov 15, 2023 - 14:32 UTC

Services have been fully operational for the last 3 hours. This incident is resolved.
Posted Nov 15, 2023 - 13:09 UTC
We have fully recovered the availability of all services. All affected environments should now be working as expected. We're monitoring the situation closely.
Posted Nov 15, 2023 - 09:47 UTC
Data reallocation is still ongoing, affecting some environments. Affected environments can experience reduce input/output performance.
Posted Nov 15, 2023 - 07:57 UTC
We are continuing to monitor the data reallocation process. Changes have been implemented to reduce the performance impact of the ongoing procedures, but the region will continue experiencing degraded performance until this is complete. We currently estimate that the ongoing process may still take a number of hours to complete fully.
Posted Nov 15, 2023 - 05:18 UTC
Routine Ceph OSDs being added into the region is currently resulting in an elevated level of Input/Output (IO) and now throttling above normal levels.
Posted Nov 15, 2023 - 04:24 UTC
We have detected an issue affecting service on the region. Our Operations team has been notified and is currently working to restore service. Projects on affected regions may experience degraded performance.
We will update you as soon as we have further information.
Posted Nov 15, 2023 - 04:17 UTC
This incident affected: Europe (Germany) (