Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-September-1 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday September 1st 2022, 16:00 - 17:57 UTC

What happened:

New test requests in all regions began to queue up in “new job” status waiting for the system to allocate them to test resources. The spike in queued test requests was detected by our internal alerting system. 

Why it happened:

An application deployment introduced a bug that caused an unrecoverable error in the resource allocation process. 

How we fixed it:

We reverted the deployment to the previous version. 

What we are doing to prevent it from happening again:

We are evaluating ways to better detect an issue like this through better test coverage in our development process.

Posted Sep 05, 2022 - 09:34 UTC

Resolved
This incident has been resolved.
Posted Sep 01, 2022 - 18:07 UTC
Update
We have taken corrective steps and all tests are running as expected in all datacenters. All services are fully operational.
Posted Sep 01, 2022 - 17:57 UTC
Update
Jobs in our APAC and EU Central datacenters have recovered. Jobs in US West and US East datacenter are currently recovering.
Posted Sep 01, 2022 - 16:54 UTC
Investigating
We are seeing test jobs failing to start in APAC, EU Central, US West, US East datacenters. We are investigating.
Posted Sep 01, 2022 - 16:36 UTC
This incident affected: Live Real Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central), Native Framework Mobile App Testing (US-West, EU-Central), Sauce Labs REST API (US-West, US-East (Headless), EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Browser Testing (US-West, US-East (Headless), EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Browser Testing (US-West, EU-Central), and Visual Testing (Visual Testing Hub, Visual Testing Infrastructure, Visual Testing REST API, Legacy Visual Testing (Screener) UI).