Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-August-26 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday August 26th 2022, 14:58 - 16:07 UTC

What happened:

In our US-West-1 region we had a brief interruption where Sauce Connect tunnels were failing to start. 

Why it happened:

The user authentication service that Sauce Connect relies on was unresponsive for approximately 5 minutes. The API gateway that serves this service was not evenly distributing requests which resulted in some pods receiving a bulk of the requests and becoming CPU bound. This caused a service within Sauce Connect to go into a crash loop and the pods left in the pool were unable to serve all the requests. 

How we fixed it:

We restarted the authentication service to evenly distribute the load. Once that happened the affected service within Sauce Connect became healthy again. 

What we are doing to prevent it from happening again:

We have added additional monitoring and alerting to inform us when the authentication service gets into this state. We are also looking at ways to better distribute requests as well as ensuring that we have the right level of resources allocated for both Sauce Connect and the authentication service.

Posted Sep 22, 2022 - 14:13 UTC

Resolved
After taking remedial action, tunnels in the US West datacenter are able to be created. Some existing tunnels may have been closed as a result. All services are fully operational.
Posted Aug 26, 2022 - 16:17 UTC
Investigating
We are seeing errors when starting Sauce Connect tunnels in the US West datacenter. We are investigating.
Posted Aug 26, 2022 - 15:56 UTC
This incident affected: Sauce Connect (US-West).