Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-October-03 Resolved Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Monday October 3rd 2022, 11:56 - 12:20 UTC

What happened:

Real Device testing and the app storage service were unavailable in both the EU and US data centers during the duration of the incident. 

Why it happened:

The service responsible for testing on real devices was not able to decode the JSON web token (JWT) issued by our authentication service, because the deserialization framework used in the JWT library did not accept JWT payloads with unknown fields. 

A deployment of the authentication service introduced a new key to the JWT structure and the RDC and app storage services involved threw an error instead of ignoring the unknown fields. 

How we fixed it:

We performed a rollback of auth-service to the previous version which removed the field from the JWT. 

What we are doing to prevent it from happening again:

We deployed fixes to the impacted services to ignore unknown fields in the JWT instead of throwing an error. We are also looking into better integration testing and API versioning as ways to better manage this in the future.

Posted Oct 19, 2022 - 16:18 UTC

Resolved
At noon UTC the user interface, ability to test Real Devices, and Sauce Storage went down. This impacted EU-Central-1 and US-West-1. During this time apps couldn't be uploaded or downloaded, saucelabs.com would not load, and tests did not run successfully. We took remedial action and all systems were operational at 12:21 UTC.
Posted Oct 03, 2022 - 11:00 UTC