Dedicated Enterprise routes issue on deployment

Incident Report for Platform.sh

Postmortem

What happened

An update to our API introduced a bug that caused the routes configuration to break when a deployment was performed.

Customer Impact

The impact was limited to a small number of projects due to the issue only occurring after a deployment which then caused projects to return a 404 or 50X error response. Affected project owners have been notified via support tickets advising the corrective actions taken by our team.

What was done to resolve the incident

We manually fixed affected projects while working on a global fix. Once the fix was ready and verified, we rolled it out to the affected projects.

Posted Nov 14, 2023 - 06:40 UTC

Resolved

This incident has been resolved.

Posted Nov 14, 2023 - 06:39 UTC

Monitoring

We have rolled out a fix and we are actively monitoring the situation. All impacted projects have undergone necessary corrections. Should any persisting issues arise, we kindly request that you provide an update through the designated ticket we have created.

Posted Nov 14, 2023 - 04:02 UTC

Identified

We have identified the issue as a result of a recent update to our API (24:35).

We have identified all potentially impacted customers and have opened support tickets for each project that did receive this update. So far we have only received reports from two customers related to this issue. We opened an incident promptly to avoid any further impact.

If you have not received a ticket notifying you of the issue you are safe to proceed as normal.

If you have any further questions do not hesitate to open a support ticket.

Posted Nov 13, 2023 - 18:44 UTC

Update

At this time there are no updates to provide.

Our teams are continuing the investigation.

Posted Nov 13, 2023 - 17:02 UTC

Update

We are continuing our investigation into today’s broken routes for dedicated environments.

At this time we have only received a minimum number of site down notifications. Our HighSLA monitoring will continue to monitor for uptime and we will proceed to reach out if your environment is impacted.

We are working to identify only the impacted dedicated clusters and will communicate once we have finished this analysis.

At this time, please refrain from deployments to your dedicated production environments as a precautionary measure.

Posted Nov 13, 2023 - 15:02 UTC

Investigating

We have detected an issue affecting our Dedicated Enterprise clusters.

This will cause your environment's routes to be incorrectly generated. Please do not deploy any Dedicated Enterprise clusters.

Our Operations and Engineering Teams are working to identify and correct the issue as soon as possible.

Posted Nov 13, 2023 - 13:42 UTC

This incident affected: Dedicated Enterprise.