Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-October-3 Service Incident
Incident Report for Sauce Labs


Monday October 2nd 2023, 11:51 UTC - Wednesday October 4th 15:20 UTC

What happened:

A small percentage of customer Appium tests failed when certain commands were executed, like capturing screenshots or executing custom scripts. 

Why it happened:

A connection pool used by our Appium server was intermittently exhausted during spikes in the usage of mid-session install scripts. During these spikes, we were leaking HTTP client connections, which caused timeouts and eventual connection issues to the Appium server. 

How we fixed it:

We restored the functionality of the service in two ways: 

  • Firstly, we gradually recycled pods to see if cleanly starting these processes would clear the issue. 
  • That bought us time to further debug the issue and we discovered the leak of HTTP client connections which led to this timeout issue. The team then put together a fix for the issue and deployed it to production. 

What we are doing to prevent it from happening again:

During the incident, we addressed the HTTP client connection leak and enhanced monitoring to better identify this specific issue should it happen again in the future.

Posted Oct 23, 2023 - 12:06 UTC

We have resolved the issue which caused timeouts when running iOS and Android real device tests in the US West and EU Central datacenters. All services are fully operational.
Posted Oct 03, 2023 - 18:33 UTC
We are seeing an increase in timeouts when running iOS and Android real device tests in the US West and EU Central datacenters. We are investigating.
Posted Oct 03, 2023 - 15:03 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).