Sauce Labs Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.

2022-May-02 Degraded performance
Incident Report for Sauce Labs
Postmortem

Dates:

Monday May 2nd 2022, 06:42 - 07:42 UTC

What happened:

Tests running in our US-East-1 datacenter were unable to be loaded in our UI.

Why it happened:

Requests to the jobs REST API endpoint were failing, which resulted in responding with 504 HTTP status code. This was caused by a hung internal DB query, which resulted in a retry mechanism kicking in, increasing load on the DB which resulted in cascading failure of subsequent queries to the jobs endpoint.

How we fixed it:

We canceled the hanging queries.

What we are doing to prevent it from happening again:

We are going to put in place a safety mechanism to prevent long running queries from overwhelming the DB as well as improving alerting to notify us earlier if a similar situation occurs.

Posted May 06, 2022 - 15:16 UTC

Resolved
We have identified the issue and taken remedial action. All services are fully operational.
Posted May 02, 2022 - 23:03 UTC
Investigating
We are currently experiencing issues loading running jobs in our Headless Cloud (us-east-1 data center) in Sauce Labs WEB UI. We're investigating.
Posted May 02, 2022 - 22:53 UTC
This incident affected: Sauce Labs REST API (US-East (Headless)), Automated Browser Testing (US-East (Headless)), and Sauce Labs UI (US-East (Headless)).