Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2024-March-27 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday March 27th 2024, 10:20 - 11:55 UTC
What happened:
Availability of real devices running iOS 14.x - 16.x was decreased, which may have resulted in an inability to obtain a device during periods of increased usage.
Why it happened:
A third-party library was updated. This update introduced a minor bug that exposed another non-recoverable bug, resulting in specific versions of iOS appearing offline in our system.
How we fixed it:
We rolled back the library update.
What we are doing to prevent it from happening again:
We are fixing the triggered defect in our code as well as improving the detection of offline devices during deployment of the affected software.
Resolved
After taking remedial action, iOS Real devices are available in all our data centers. This incident is now resolved.
Investigating
We are experiencing reduced availability of iOS Real devices older than iOS 17 in all our data centers. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central, US-East) and Live Real Device Testing (US-West, EU-Central, US-East).