Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-September-12 Resolved Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Tuesday September 12th 2023, 16:03 - 16:29 UTC

What happened:

Customers could not start tests in the EU-Central region and, after 15 minutes, in the US-West region. 

Why it happened:

Tests failed to start due to an issue with starting the virtual machines they run on. The service responsible for creating the virtual machines had a bug that introduced a cyclical dependency, which prevented it from starting correctly.

How we fixed it:

To restore normal operation, we performed the following:

  1. Deployed the previous known working version of the affected service to both EU and US data centers
  2. Monitored to ensure that tests stopped timing out

What we are doing to prevent it from happening again:

We addressed the bug, tested it, and released the original intended change back in Production.

Posted Sep 28, 2023 - 15:02 UTC

Resolved
We experienced intermittent issues starting live and automated tests on our Virtual Cloud between 1600 UTC and 1630 UTC in our EU Central Data Center and between 1615 UTC and 1630 UTC in our US West Data Center. This issue is now resolved, all services are fully operational.
Posted Sep 12, 2023 - 14:00 UTC