Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2023-April-29 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Saturday April 29th 2023, 07:03 - 09:06 UTC

What happened:

Customers experienced elevated wait times and error rates for tests running on macOS and iOS simulators for the duration of the incident.

Why it happened:

A third-party automated upgrade erroneously halted services supporting service discovery for one of our data centers. Without service discovery, VMs were not able to be restarted.

How we fixed it:

We restarted the service discovery services, and normal operation resumed.

What we are doing to prevent it from happening again:

We are preventing live upgrades of the service in question, moving it to our regular release workflow.

Posted May 09, 2023 - 13:07 UTC

Resolved

After taking remedial action, MacOS Desktop and iOS Virtual mobile device test wait times and error rates in our US-West-1 Data center have returned to normal levels. This issue is resolved.
Posted Apr 29, 2023 - 09:39 UTC

Monitoring

After taking remedial action, MacOS Desktop and iOS Virtual mobile device tests wait times and error rates in our US-West-1 Data center are returning to normal levels. We are currently monitoring.
Posted Apr 29, 2023 - 09:08 UTC

Investigating

MacOS Desktop and iOS Virtual mobile device tests are seeing long wait times and a high rate of errors up in our US-West-1 Data center. We are investigating.
Posted Apr 29, 2023 - 08:08 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), and Live Virtual Mobile Device Testing (US-West).