Outage on Dedicated Infrastructure
Incident Report for Platform.sh
Postmortem

We apologize for the issues your team experienced from this event and understand the impact this has had on you and your stakeholders. The Platform.sh On-Call team was notified of HTTP 502 errors via our monitoring system on September 8th at 13:52 UTC. Our team began investigating and identified that we inadvertently deployed an expired TLS certificate that was applied to the origin servers at 13:48 UTC. This expired certificate caused HTTP requests to the origin to produce HTTP 502 responses. At 14:03 UTC, we verified and reverted the expired TLS certificate to the correct version. Once the deployment was reverted, the service was restored. We officially closed this event at 15:16 UTC.

Timeline:

  • 2020-09-08 13:48 Inadvertently deployed expired origin certificate.
  • 2020-09-08 13:52 Platform.sh monitoring alerted with HTTP 502 errors.
  • 2020-09-08 14:03 We noticed that the old version of the origin certificate was deployed and reverted it.
  • 2020-09-08 14:08 The status page was updated with a new incident created
  • 2020-09-08 15:16 Incident is closed

Remaining Issues or Risks:

  • None.

Actions Taken:

  • We have implemented more stringent change control processes to account for similar updates and to prevent a future occurrence.
Posted Sep 11, 2020 - 09:43 UTC

Resolved
This incident has been resolved.
Posted Sep 08, 2020 - 15:17 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 08, 2020 - 14:38 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 08, 2020 - 14:18 UTC
Investigating
We have detected an issue affecting service on the Dedicated Infrastructure. We are currently working to restore service.
This issue affects multiple production sites as well as staging environments. Access to your site may be affected.

We will update you as soon as we have further information.
Posted Sep 08, 2020 - 14:10 UTC
This incident affected: Dedicated Enterprise.