Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-April-24 Service Incident
Incident Report for Sauce Labs


Monday April 24th 2023, 15:35 - 16:40 UTC

What happened:

During the incident, the percentage of real devices in an error state increased to about 20%. 

  • The majority impacted devices were running Android 12 and 13
  • Android 11 and below remained in a healthy state
  • Private devices were also impacted
  • The devices only entered an errored state after running a test

Why it happened:

Android toolkit had to be compiled with the latest stable SDK to be able to run on Android 14 beta 1. With the latest SDK, it’s no longer possible to call the hidden system API through reflection with the current configuration. Due to this change, the Android toolkit failed to clean the device Locale, which is a mandatory cleaning step. If that step fails, then the device immediately goes into an error state. 

How we fixed it:

We rolled back the Android toolkit to an older version compiled with an older SDK.

What we are doing to prevent it from happening again:

We are looking into ways to handle device cleaning better in these scenarios, for example, in some cases cleaning device locale isn’t always needed. We are also improving how we test and detect this type of issue throughout our development lifecycle.

Posted May 17, 2023 - 10:16 UTC

After taking remedial action, device availability is back to normal in both US and EU datacenters. All services are fully operational.
Posted Apr 24, 2023 - 16:48 UTC
We are currently experiencing decreased availability in our Real Device Cloud for both EU and US datacenters. We are currently investigating.
Posted Apr 24, 2023 - 16:30 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).