Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2022-October-03 Resolved Service Incident
Incident Report for Sauce Labs


Monday October 3rd 2022, 11:56 - 12:20 UTC

What happened:

Real Device testing and the app storage service were unavailable in both the EU and US data centers during the duration of the incident. 

Why it happened:

The service responsible for testing on real devices was not able to decode the JSON web token (JWT) issued by our authentication service, because the deserialization framework used in the JWT library did not accept JWT payloads with unknown fields. 

A deployment of the authentication service introduced a new key to the JWT structure and the RDC and app storage services involved threw an error instead of ignoring the unknown fields. 

How we fixed it:

We performed a rollback of auth-service to the previous version which removed the field from the JWT. 

What we are doing to prevent it from happening again:

We deployed fixes to the impacted services to ignore unknown fields in the JWT instead of throwing an error. We are also looking into better integration testing and API versioning as ways to better manage this in the future.

Posted Oct 19, 2022 - 16:18 UTC

At noon UTC the user interface, ability to test Real Devices, and Sauce Storage went down. This impacted EU-Central-1 and US-West-1. During this time apps couldn't be uploaded or downloaded, would not load, and tests did not run successfully. We took remedial action and all systems were operational at 12:21 UTC.
Posted Oct 03, 2022 - 11:00 UTC