Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2022-August-5 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday August 4th 2022, 23:39 - August 5th 00:34 UTC

What happened:

We deployed an enhancement to support the latest version of Geckodriver for FireFox based tests. After the deployment we noticed an increase in error rate for tests running on FireFox. 

Why it happened:

This happened because some customers were using a remote debugging webdriver capability in FireFox that was no longer supported in the latest version of Geckodriver. 

How we fixed it:

We rolled back the deployment. 

What we are doing to prevent it from happening again:

When deploying changes that require our customers to make changes to their configuration or tests we will increase our communication and ensure that they are aware of the changes.

Posted Aug 22, 2022 - 11:43 UTC

Resolved

After taking remedial action, error rates on Windows, which were specific to Firefox testing, have decreased to normal levels. All services are fully operational.
Posted Aug 05, 2022 - 01:51 UTC

Update

We’ve seen storage error rates go down to normal levels but we are seeing high error rates on Windows both on our US and EU data centers. We are investigating.
Posted Aug 05, 2022 - 00:04 UTC

Investigating

We are seeing an elevated number of errors on our storage service affecting our Virtual and Real Device Cloud as well as any saucectl test in our US Data Center. We are investigating.
Posted Aug 04, 2022 - 23:27 UTC
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), and Native Framework Mobile App Testing (US-West).