Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2024-July-4 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday July 4th, 2024 13:13 - 14:06 UTC 

What happened:

An incident occurred due to I/O errors on the database node, causing connectivity issues within our infrastructure. This affected Virtual jobs and Tunnel services in the US-West region, leading to disruptions in Sauce Labs products.

Why it happened:

Our job orchestration service failed to connect to a backend data store.

How we fixed it:

The data store issues were resolved, restoring connectivity.

What we are doing to prevent it from happening again:

We’re undertaking several initiatives including process improvements and enhanced monitoring, outlining steps to fix the issue and prevent it from happening again,

Posted Sep 23, 2024 - 10:05 UTC

Resolved

After taking remedial action, all services are operating as normal. This incident is resolved.
Posted Jul 04, 2024 - 14:28 UTC

Update

We are currently seeing elevated error rates in Desktop and Virtual Mobile tests using the US-West-1 datacenter. We are currently investigating.
Posted Jul 04, 2024 - 14:05 UTC

Investigating

All jobs in our US-West-1 Data Center are failing to start. We are investigating.
Posted Jul 04, 2024 - 13:49 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), and Native Framework Mobile App Testing (US-West).