Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2021-Dec-02 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday December 2nd 2021, 2:00 - 4:30 UTC
What happened:
Android devices in the US-West-1 data center failed during regularly recurring health checks and became disabled.
Why it happened:
Two services, one responsible for running health checks on our real devices, the other responsible for installing Android applications were installed on the same VM. The installation service has a disk cache that used most of the disk space and impacted the functionality of the service running the health checks. The devices that were checked failed the health check and they were disabled for customer tests.
How we fixed it:
The services were restarted.
What we are doing to prevent it from happening again:
The two services were separated into different VMs. Also the time-to-live of the files in the installation cache was reduced.
Resolved
This incident has been resolved.
Investigating
We are seeing an elevated number of unavailable real devices in our US-West data center. We are investigating.
This incident affected: Live Real Device Testing (US-West) and Automated Real Device Testing (US-West).