Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-August-26 Service Incident (2)
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday August 26th 2022, 14:00 - 15:56 UTC
What happened:
Tests requesting MacOS and iOS resources could not start due to network link saturation.
Why it happened:
A large number of Apps were downloading directly from Apple which saturated the networking link in our Mac virtualization cloud.
How we fixed it:
The large volume of Apps downloading subsided on its own.
What we are doing to prevent it from happening again:
We are continuing to review the provider’s network architecture to uncover other improvement areas to increase bandwidth within their network. We have also worked with them to gain deeper insight into their network performance monitoring and alerting.
In addition, we are looking into options to improve the high availability and performance of our macOS virtualization cloud.
Resolved
All tests are clear and running as expected on both iOS and Mac in the US West datacenter. All services are fully operational.
Investigating
We are seeing Mac and iOS mobile tests failing to start in the US West datacenter.
App Storage is affected. We are seeing high wait times. We are investigating.
App Storage is affected. We are seeing high wait times. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), and Visual Testing (Visual Testing Hub).