Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2022-August-26 Service Incident
Incident Report for Sauce Labs


Friday August 26th 2022, 14:58 - 16:07 UTC

What happened:

In our US-West-1 region we had a brief interruption where Sauce Connect tunnels were failing to start. 

Why it happened:

The user authentication service that Sauce Connect relies on was unresponsive for approximately 5 minutes. The API gateway that serves this service was not evenly distributing requests which resulted in some pods receiving a bulk of the requests and becoming CPU bound. This caused a service within Sauce Connect to go into a crash loop and the pods left in the pool were unable to serve all the requests. 

How we fixed it:

We restarted the authentication service to evenly distribute the load. Once that happened the affected service within Sauce Connect became healthy again. 

What we are doing to prevent it from happening again:

We have added additional monitoring and alerting to inform us when the authentication service gets into this state. We are also looking at ways to better distribute requests as well as ensuring that we have the right level of resources allocated for both Sauce Connect and the authentication service.

Posted Sep 22, 2022 - 14:13 UTC

After taking remedial action, tunnels in the US West datacenter are able to be created. Some existing tunnels may have been closed as a result. All services are fully operational.
Posted Aug 26, 2022 - 16:17 UTC
We are seeing errors when starting Sauce Connect tunnels in the US West datacenter. We are investigating.
Posted Aug 26, 2022 - 15:56 UTC
This incident affected: Sauce Connect (US-West).