Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-Dec-01 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday December 1st 2021, 02:40 - 04:24 UTC

What happened:

Tests running on  iOS simulators, Android emulators, and desktop browser/OSs in our US-West and EU-Central data centers experienced high wait times and errors.

Why it happened:

Unforeseen issues in our deployment flow caused the cloud capacity to become unavailable. Safeguards were not in place to prevent this from rolling out to both the EU and US.

How we fixed it:

Services were restarted to restore capacity. Some Mac/iOS tests were terminated to expedite recovery in the US-West data center.

What we are doing to prevent it from happening again:

We are introducing safeguards and adding additional checks in our services to increase resiliency while at the same time working with our cloud providers to improve traffic flows.

Posted Dec 09, 2021 - 16:02 UTC

Resolved
iOS Simulators and MacOS Desktop tests have fully recovered in US Data Center. All services are fully operational.
Posted Dec 01, 2021 - 04:29 UTC
Investigating
Our US and EU Regions experienced high wait times and errors between 02:40 - 03:10 UTC and have since recovered.
iOS Simulators and MacOS Desktop tests are still experiencing high wait times in US Data Center. During this time, tests may fail to start. We are taking remedial action.
Posted Dec 01, 2021 - 03:46 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), and Native Framework Mobile App Testing (US-West).