Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2022-May-02 Degraded performance
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday May 2nd 2022, 06:42 - 07:42 UTC
What happened:
Tests running in our US-East-1 datacenter were unable to be loaded in our UI.
Why it happened:
Requests to the jobs REST API endpoint were failing, which resulted in responding with 504 HTTP status code. This was caused by a hung internal DB query, which resulted in a retry mechanism kicking in, increasing load on the DB which resulted in cascading failure of subsequent queries to the jobs endpoint.
How we fixed it:
We canceled the hanging queries.
What we are doing to prevent it from happening again:
We are going to put in place a safety mechanism to prevent long running queries from overwhelming the DB as well as improving alerting to notify us earlier if a similar situation occurs.
Resolved
We have identified the issue and taken remedial action. All services are fully operational.
Investigating
We are currently experiencing issues loading running jobs in our Headless Cloud (us-east-1 data center) in Sauce Labs WEB UI. We're investigating.
This incident affected: Automated Browser Testing (US-East (Headless)), Sauce Labs REST API (US-East (Headless)), and Sauce Labs Dashboard (US-East (Headless)).