Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-May-09 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday May 9th 2022, 15:00 - 17:00 UTC
What happened:
Tests running in our US-East-1 data center (Headless Cloud) were failing to start.
Why it happened:
The cause of this issue was a bug in our user service which returned an error message when trying to allocate a Sauce Connect tunnel for new tests. This caused a cascading situation where new tests were unable to start and resources to start further tests were exhausted.
How we fixed it:
We scaled down services and removed all running tests and cleaned the queue of waiting tests, in order to free up resources.
What we are doing to prevent it from happening again:
We plan on changing the in order in which we reserve resources within the system in order to prevent the cascading error condition that can starve resources when tunnels cannot be allocated.
Resolved
After taking remedial action all tests are clear and running as expected. All services are fully operational.
Identified
We have identified the issue causing tests in our us-east-1 data center (Headless Cloud) to fail. We are taking remedial action.
Investigating
Tests running in our us-east-1 data center (Headless Cloud) are failing to start. We are currently investigating.