2023-February-1 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday February 1st 2023, 12:25 - 13:55 UTC

What happened:

Customers experienced elevated wait times for virtual devices running a specific macOS version for the duration of the incident.

Why it happened:

An unusual spike in demand for a particular macOS version caused starvation for this virtual device type. The Mac cloud running macOS and iOS suffers from device starvation when there are large spikes in demand. In this particular case, we saw not only a spike in requests for devices in our Mac cloud but also for a specific version of macOS which led to elevated wait times.

How we fixed it:

During these incidents, we can only wait for our Mac cloud to catch up with the demand.

What we are doing to prevent it from happening again:

We continue to look at ways to increase our Mac cloud in both our US and EU regions. In the short term, we are rebalancing the OS versions we have pre-booted and ready to be allocated to tests based on recent usage.

Posted Feb 23, 2023 - 21:48 UTC

Resolved

Wait times have returned to normal levels. This incident is resolved.

Posted Feb 01, 2023 - 13:42 UTC

Investigating

We are experiencing elevated wait times for MacOS and iOS tests in our EU datacenter, which may cause tests to be slow to start. We are investigating.

Posted Feb 01, 2023 - 13:06 UTC

This incident affected: Automated Virtual Mobile Device Testing (EU-Central), Live Browser Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), and Automated Browser Testing (EU-Central).