Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-May-24 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday May 24 2022, 17:10 - 22:55 UTC
What happened:
iOS simulator tests running in US-West experienced an elevated rate of test errors.
Why it happened:
A deployment combined with a high rate of iOS tests downloading applications caused bandwidth contention on our Mac infrastructure. iOS applications failed to download, causing test errors.
How we fixed it:
The deployment was halted, alleviating bandwidth contention.
What we are doing to prevent it from happening again:
- Deployment bandwidth limits are being lowered to reduce the chance of impact on tests.
- Mac infrastructure is being upgraded to increase bandwidth capacity.
Resolved
After taking remedial action we have seen the error rates come back to normal levels. iOS tests are running as expected. All services are fully operational.
Investigating
We have received multiple reports on a variety of errors seen in our iOS Simulators in our US-West-1 datacenter. We are investigating.
This incident affected: Automated Virtual Mobile Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), and Native Framework Mobile App Testing (US-West).