Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2021-Dec-01 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday December 1st 2021, 02:40 - 04:24 UTC
What happened:
Tests running on iOS simulators, Android emulators, and desktop browser/OSs in our US-West and EU-Central data centers experienced high wait times and errors.
Why it happened:
Unforeseen issues in our deployment flow caused the cloud capacity to become unavailable. Safeguards were not in place to prevent this from rolling out to both the EU and US.
How we fixed it:
Services were restarted to restore capacity. Some Mac/iOS tests were terminated to expedite recovery in the US-West data center.
What we are doing to prevent it from happening again:
We are introducing safeguards and adding additional checks in our services to increase resiliency while at the same time working with our cloud providers to improve traffic flows.
Resolved
iOS Simulators and MacOS Desktop tests have fully recovered in US Data Center. All services are fully operational.
Investigating
Our US and EU Regions experienced high wait times and errors between 02:40 - 03:10 UTC and have since recovered.
iOS Simulators and MacOS Desktop tests are still experiencing high wait times in US Data Center. During this time, tests may fail to start. We are taking remedial action.
iOS Simulators and MacOS Desktop tests are still experiencing high wait times in US Data Center. During this time, tests may fail to start. We are taking remedial action.
This incident affected: Native Mobile App Testing (US-West), Live Virtual Mobile Device Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Browser Testing (US-West), and Live Browser Testing (US-West).