Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2022-May-02 Degraded performance

Incident Report for Sauce Labs

Postmortem

Dates:

Monday May 2nd 2022, 06:42 - 07:42 UTC

What happened:

Tests running in our US-East-1 datacenter were unable to be loaded in our UI.

Why it happened:

Requests to the jobs REST API endpoint were failing, which resulted in responding with 504 HTTP status code. This was caused by a hung internal DB query, which resulted in a retry mechanism kicking in, increasing load on the DB which resulted in cascading failure of subsequent queries to the jobs endpoint.

How we fixed it:

We canceled the hanging queries.

What we are doing to prevent it from happening again:

We are going to put in place a safety mechanism to prevent long running queries from overwhelming the DB as well as improving alerting to notify us earlier if a similar situation occurs.

Posted May 06, 2022 - 15:16 UTC

Resolved

We have identified the issue and taken remedial action. All services are fully operational.
Posted May 02, 2022 - 23:03 UTC

Investigating

We are currently experiencing issues loading running jobs in our Headless Cloud (us-east-1 data center) in Sauce Labs WEB UI. We're investigating.
Posted May 02, 2022 - 22:53 UTC