2021-March-8 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Monday, March 8, 18:39 - 23:30 UTC

What happened:

We were seeing elevated wait times for automated and manual MacOS tests in our US-West-1 data center. Customers may have seen MacOS and iOS Simulator tests taking a long time to start, or failing to start. Windows, Linux and Android tests were unaffected.

Why it happened:

Due to high sustained peak usage combined with intensive CPU operations from a customer application our Mac Cloud CPU performance became degraded.

How we fixed it:

Additional data center caching servers were deployed to improve performance and distribution of application storage. We also implemented throttling of targeted jobs to reduce the CPU load on our US-West Mac Cloud.

What we are doing to prevent it from happening again:

In the future more aggressive throttling will be implemented on a per customer basis to keep the performance of our services within established baselines. A Sev1 issue will be opened with affected customers to work with them to resolve the issue specific to their usage.

Posted Mar 16, 2021 - 11:11 UTC

Resolved

This incident has been resolved. All services are fully operational.

Posted Mar 09, 2021 - 13:25 UTC

Monitoring

Wait times have returned to normal levels. We are continuing to monitor the situation.

Posted Mar 09, 2021 - 11:55 UTC

Update

Wait times continue to be elevated for automated and manual MacOS and iOS Simulator tests in our US-West-1 Data Center. We are still investigating.

Posted Mar 09, 2021 - 09:15 UTC

Update

We are seeing elevated wait times for automated and manual MacOS tests, including iOS Simulators, in the US-West Data Centre only. Customers may see tests taking a long time to start, or failing to start. Windows, Linux and Android tests are unaffected. We are investigating.

Posted Mar 09, 2021 - 07:16 UTC

Investigating

We are seeing elevated wait times for automated and manual MacOS tests, including iOS Simulators. Customers may see tests taking a long time to start, or failing to start. Windows, Linux and Android tests are unaffected. We are investigating.

Posted Mar 09, 2021 - 07:15 UTC

This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Live Virtual Mobile Device Testing (US-West).