Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-August-5 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday August 4th 2022, 23:39 - August 5th 00:34 UTC
What happened:
We deployed an enhancement to support the latest version of Geckodriver for FireFox based tests. After the deployment we noticed an increase in error rate for tests running on FireFox.
Why it happened:
This happened because some customers were using a remote debugging webdriver capability in FireFox that was no longer supported in the latest version of Geckodriver.
How we fixed it:
We rolled back the deployment.
What we are doing to prevent it from happening again:
When deploying changes that require our customers to make changes to their configuration or tests we will increase our communication and ensure that they are aware of the changes.
Resolved
After taking remedial action, error rates on Windows, which were specific to Firefox testing, have decreased to normal levels. All services are fully operational.
Update
We’ve seen storage error rates go down to normal levels but we are seeing high error rates on Windows both on our US and EU data centers. We are investigating.
Investigating
We are seeing an elevated number of errors on our storage service affecting our Virtual and Real Device Cloud as well as any saucectl test in our US Data Center. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), and Native Framework Mobile App Testing (US-West).