Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-July-2 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday, July 2, 10:54 - 15:57 UTC
What happened?
Some customers received erroneous messages stating they were running at their maximum concurrency, preventing them from starting new tests.
Why did it happen?
During a routine deployment, a serialization library was updated. The update contained a dependency with a non-compatible, breaking change that made the read/write values of this service to not work.
How was it resolved?
The deployment was rolled back.
What are we doing to prevent it from happening again?
We have deployed an upgrade which removed the dependency on the library that caused this issue. We will also be improving our post release external monitoring system to monitor the API endpoint that was affected by this issue.
Resolved
We have taken remedial action. CCY endpoints and tests now should run as expected. All services are fully operational.
Investigating
We are getting reports on incorrect CCY usage on our UI. This may cause tests to fail incorrectly due to too many VMs running. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), and Sauce Labs REST API (US-West, EU-Central).