Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-May-5 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Friday May 5th 2023, 10:05 - 11:02 UTC

What happened:

For the duration of the incident, customers experienced slow performance on the Sauce Labs UI, and when executing test requests in all regions.

Why it happened:

A database node became responsive due to a higher than normal number of requests. When this happens, the database failover mechanism typically fires but it failed to do so fully due to a configuration issue.

How we fixed it:

The faulty database node was manually disabled. As a result, all services connected to another node.

What we are doing to prevent it from happening again:

The failover mechanism has been fixed. Next time a similar situation happens, our services will be able to automatically switch to a secondary database node.

Posted May 26, 2023 - 17:39 UTC

Resolved

After taking remedial action, access to the Sauce Labs platform UI has been fully restored, and tests and Sauce Connect tunnels are working across all Data Centers. All services are fully operational.
Posted May 05, 2023 - 11:13 UTC

Investigating

We are currently experiencing slow performance on the Sauce Labs UI and we are seeing errors when starting tests and Sauce Connect tunnels across all Data Centers. We are investigating.
Posted May 05, 2023 - 10:39 UTC
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central, US-East), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central, US-East), Sauce Connect (US-West, EU-Central), Sauce Labs REST API (US-West, EU-Central), Sauce Labs Dashboard (US-West, EU-Central), Insights (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).