Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-September-12 Resolved Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday September 12th 2023, 16:03 - 16:29 UTC
What happened:
Customers could not start tests in the EU-Central region and, after 15 minutes, in the US-West region.
Why it happened:
Tests failed to start due to an issue with starting the virtual machines they run on. The service responsible for creating the virtual machines had a bug that introduced a cyclical dependency, which prevented it from starting correctly.
How we fixed it:
To restore normal operation, we performed the following:
- Deployed the previous known working version of the affected service to both EU and US data centers
- Monitored to ensure that tests stopped timing out
What we are doing to prevent it from happening again:
We addressed the bug, tested it, and released the original intended change back in Production.
Resolved
We experienced intermittent issues starting live and automated tests on our Virtual Cloud between 1600 UTC and 1630 UTC in our EU Central Data Center and between 1615 UTC and 1630 UTC in our US West Data Center. This issue is now resolved, all services are fully operational.