Sauce Labs Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2022-May-02 Degraded performance
Incident Report for Sauce Labs


Monday May 2nd 2022, 06:42 - 07:42 UTC

What happened:

Tests running in our US-East-1 datacenter were unable to be loaded in our UI.

Why it happened:

Requests to the jobs REST API endpoint were failing, which resulted in responding with 504 HTTP status code. This was caused by a hung internal DB query, which resulted in a retry mechanism kicking in, increasing load on the DB which resulted in cascading failure of subsequent queries to the jobs endpoint.

How we fixed it:

We canceled the hanging queries.

What we are doing to prevent it from happening again:

We are going to put in place a safety mechanism to prevent long running queries from overwhelming the DB as well as improving alerting to notify us earlier if a similar situation occurs.

Posted May 06, 2022 - 15:16 UTC

We have identified the issue and taken remedial action. All services are fully operational.
Posted May 02, 2022 - 23:03 UTC
We are currently experiencing issues loading running jobs in our Headless Cloud (us-east-1 data center) in Sauce Labs WEB UI. We're investigating.
Posted May 02, 2022 - 22:53 UTC
This incident affected: Automated Browser Testing (US-East (Headless)), Sauce Labs REST API (US-East (Headless)), and Sauce Labs Dashboard (US-East (Headless)).