Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-February-2 Service Incident 3
Incident Report for Sauce Labs


Thursday February 2nd 2023, 21:20 - 23:15 UTC

What happened:

The number of devices reported as unavailable in one of our US data centers reached 5% for iOS devices and 20%. Our data center engineers were alerted when device availability dropped below 80% and continued to show a sustained drop in available devices. 

Why it happened:

A service called device inspector queries our device pools and reports back the set of available devices. This service runs on 2 nodes, and one of them failed to query the device pool and subsequently marked devices as unavailable. 

How we fixed it:

We restarted the affected device inspector node, and the number of offline devices returned to normal levels. 

What we are doing to prevent it from happening again:

We are looking at ways to improve the monitoring of the device inspection process and the logic it uses to handle these situations better.

Posted Feb 23, 2023 - 22:04 UTC

After taking remedial action, Real Device availability on US-West-1 data center is back at full capacity. This issue is now resolved, all services are fully operational.
Posted Feb 02, 2023 - 23:45 UTC
We are currently experiencing decreased availability in our Real Device Cloud for the US-West-1 datacenter. We are currently investigating
Posted Feb 02, 2023 - 22:50 UTC
This incident affected: Automated Real Device Testing (US-West) and Live Real Device Testing (US-West).