Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-September-22 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday September 22nd 2021, 10:40 - 13:30 UTC
What happened:
Some Android devices on the public cloud in the US-West-1 data center were not available for running tests.
Why it happened:
A number of devices had lost their connection to the internet, resulting in the devices being automatically disabled. Our automated cleaning and setup process was not successful in recovering and restoring internet connectivity.
How we fixed it:
We improved the logic that allows the system to reset the internet connection for devices.
What we are doing to prevent it from happening again:
We added additional monitoring steps to give us better data about when and how devices lose their connection to the internet, while also improving the wifi configuration inside of our data centers.
Resolved
After making changes to resolve the issue, device availability is back to normal levels. This incident is resolved.
Investigating
We are seeing an elevated proportion of unavailable Real Devices in our US-West-1 data center due to wifi issues. Customers may also be experiencing connectivity issues whilst running tests on these devices. We are investigating.
This incident affected: Automated Real Device Testing (US-West) and Live Real Device Testing (US-West).