Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-August-26 Service Incident (2)
Incident Report for Sauce Labs
Postmortem

Dates:

Friday August 26th 2022, 14:00 - 15:56 UTC

What happened:

Tests requesting MacOS and iOS resources could not start due to network link saturation.

Why it happened:

A large number of Apps were downloading directly from Apple which saturated the networking link in our Mac virtualization cloud.  

How we fixed it:

The large volume of Apps downloading subsided on its own. 

What we are doing to prevent it from happening again:

We are continuing to review the provider’s network architecture to uncover other improvement areas to increase bandwidth within their network. We have also worked with them to gain deeper insight into their network performance monitoring and alerting.

In addition, we are looking into options to improve the high availability and performance of our macOS virtualization cloud.

Posted Sep 19, 2022 - 14:40 UTC

Resolved
All tests are clear and running as expected on both iOS and Mac in the US West datacenter. All services are fully operational.
Posted Aug 26, 2022 - 19:10 UTC
Investigating
We are seeing Mac and iOS mobile tests failing to start in the US West datacenter.
App Storage is affected. We are seeing high wait times. We are investigating.
Posted Aug 26, 2022 - 18:54 UTC
This incident affected: Automated Virtual Mobile Device Testing (US-West), Automated Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), and Visual Testing (Visual Testing Hub).