Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-May-02 Degraded performance
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday May 2nd 2022, 06:42 - 07:42 UTC
What happened:
Tests running in our US-East-1 datacenter were unable to be loaded in our UI.
Why it happened:
Requests to the jobs REST API endpoint were failing, which resulted in responding with 504 HTTP status code. This was caused by a hung internal DB query, which resulted in a retry mechanism kicking in, increasing load on the DB which resulted in cascading failure of subsequent queries to the jobs endpoint.
How we fixed it:
We canceled the hanging queries.
What we are doing to prevent it from happening again:
We are going to put in place a safety mechanism to prevent long running queries from overwhelming the DB as well as improving alerting to notify us earlier if a similar situation occurs.
Resolved
We have identified the issue and taken remedial action. All services are fully operational.
Investigating
We are currently experiencing issues loading running jobs in our Headless Cloud (us-east-1 data center) in Sauce Labs WEB UI. We're investigating.