Partial Outage on de-2.platform.sh

Incident Report for Platform.sh

Postmortem

What happened

We added object storage devices to the de-2.platform.sh region. This change unexpectedly caused I/O performance issues.

Customer impact

Affected environments experienced slower than usual read and write operations, or their writable mounts suddenly became read-only.

Resolution steps

The response team added more object storage, allowing the data recalibration to complete faster, and to provide more I/O throughput to affected projects. Read-only mounts were manually investigated and recovered.

Posted Nov 15, 2023 - 14:32 UTC

Resolved

Services have been fully operational for the last 3 hours. This incident is resolved.

Posted Nov 15, 2023 - 13:09 UTC

Monitoring

We have fully recovered the availability of all services. All affected environments should now be working as expected. We're monitoring the situation closely.

Posted Nov 15, 2023 - 09:47 UTC

Update

Data reallocation is still ongoing, affecting some environments. Affected environments can experience reduce input/output performance.

Posted Nov 15, 2023 - 07:57 UTC

Update

We are continuing to monitor the data reallocation process. Changes have been implemented to reduce the performance impact of the ongoing procedures, but the region will continue experiencing degraded performance until this is complete. We currently estimate that the ongoing process may still take a number of hours to complete fully.

Posted Nov 15, 2023 - 05:18 UTC

Identified

Routine Ceph OSDs being added into the region is currently resulting in an elevated level of Input/Output (IO) and now throttling above normal levels.

Posted Nov 15, 2023 - 04:24 UTC

Investigating

We have detected an issue affecting service on the de-2.platform.sh region. Our Operations team has been notified and is currently working to restore service. Projects on affected regions may experience degraded performance.
We will update you as soon as we have further information.

Posted Nov 15, 2023 - 04:17 UTC

This incident affected: Europe (Germany) (de-2.platform.sh).