Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-October-12 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday October 12th 2022, 19:10 - 19:40 UTC

What happened:

Real device tests in our US West region failed due to connectivity issues between RDC services (test orchestration) running in GCP and the data center where our physical devices reside. These connectivity issues lasted approximately 30 minutes.

Why it happened:

It still needs to be determined where the network connectivity issue started. We are still working with Google to research the cause, and most indicators point to an issue on their end.

How we fixed it:

The network connectivity restored itself.

What we are doing to prevent it from happening again:

We are increasing the frequency we run our end-to-end tests in production to detect these incidents faster.

Posted Nov 08, 2022 - 13:49 UTC

Resolved
We have seen all real device testing stabilized for the last 60 minutes in our US-West-1 data center. All services are fully operational.
Posted Oct 12, 2022 - 20:52 UTC
Monitoring
We are seeing real device tests in the US-West-1 datacenter working as expected again. We are monitoring.
Posted Oct 12, 2022 - 20:11 UTC
Investigating
We are seeing real device tests in the US-West-1 datacenter failing. We are investigating.
Posted Oct 12, 2022 - 19:47 UTC
This incident affected: Automated Real Device Testing (US-West).