Sauce Labs Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-June-19 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Monday June 19th 2023, 15:19 - 15:56 UTC

What happened:

An infrastructure change caused devices in all RDC data centers to falsely report as busy. While ongoing sessions continued to work, all new customer sessions could not start.

Why it happened:

A change was made to RDC infrastructure, causing device cleaning to fail. This change impacted the permissions devices require to access dependencies during cleaning.

How we fixed it:

The infrastructure configuration change was reverted.

What we are doing to prevent it from happening again:

We are implementing additional guardrails to ensure the infrastructure misconfiguration is not reintroduced in the future.

We are adding additional alerting around device availability, as well as adding resiliency to device cleaning to avoid complete failure based on dependency access.

Posted Jul 07, 2023 - 11:43 UTC

Resolved
After taking remedial action, Real devices are returning to normal availability in all datacenters. This issue is now resolved
Posted Jun 19, 2023 - 16:01 UTC
Update
Real devices are showing very limited availability in the US-West-1, US-East-4 and EU-Central-1 datacenters. We are investigating.
Posted Jun 19, 2023 - 15:40 UTC
Investigating
Real devices are showing very limited availability in the US-West-1 and EU-Central-1 datacenters. We are investigating.
Posted Jun 19, 2023 - 15:37 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central, US-East), Live Real Device Testing (US-West, EU-Central, US-East), and Native Framework Mobile App Testing (US-West, EU-Central).