Environment Data Sync Not Functional

Incident Report for Platform.sh

Postmortem

WHAT HAPPENED

Here is the list of affected regions:

au.platform.sh
au-2.platform.sh
ca-1.platform.sh
uk-1.platform.sh
us.platform.sh
us-2.platform.sh

Above affected regions received recent internal upgrades having environments not functioning as expected for data synchronization activities.
This also includes new environments creation requiring cloning parent environment's data.

‌

CUSTOMER IMPACT

Customers' environments were affected for up to 24 hours and 29 minutes in total.
From 2024-01-31 06:21 UTC to 2024-02-01 06:50 UTC
(based on earliest timestamp of region received the upgrade).

‌

Existing environments including live sites should NOT be affected.

‌

WHAT WAS DONE TO RESOLVE THE INCIDENT

After extensive investigation, our engineering team implemented an emergency fix and deployed it to all the affected regions to correct that unexpected behavior.
The data synchronization should now function as expected.

Posted Feb 01, 2024 - 07:18 UTC

Resolved

This incident has been resolved.

Posted Feb 01, 2024 - 07:12 UTC

Monitoring

Our engineering team had just deployed the fix to the affected regions.
We are still monitoring the regions.

Posted Feb 01, 2024 - 06:50 UTC

Update

Our engineers had implemented the fix and now we are finalizing necessary process for testing and verification phase before releasing the fix to the affected regions.

Posted Feb 01, 2024 - 04:21 UTC

Update

We are continuing to work on a fix for this issue.

Posted Feb 01, 2024 - 02:23 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Jan 31, 2024 - 23:58 UTC

Update

We are continuing to investigate this issue.

Posted Jan 31, 2024 - 22:32 UTC

Update

We are continuing to investigate this issue.

Posted Jan 31, 2024 - 20:55 UTC

Investigating

We are seeing issues with syncing data between environments in multiple regions. Our operations team is investigating. Affected projects will not be able to synchronize data, and new environments will not contain data from their parent environment.

There is no impact to the availability of production sites.

Posted Jan 31, 2024 - 19:33 UTC

This incident affected: Australia (au.platform.sh), Australia East (au-2.platform.sh), Canada (ca-1.platform.sh), United Kingdom (uk-1.platform.sh), USA (East) (us.platform.sh), and USA-2 (East 2) (us-2.platform.sh).