Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2023-June-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday June 19th 2023, 15:19 - 15:56 UTC
What happened:
An infrastructure change caused devices in all RDC data centers to falsely report as busy. While ongoing sessions continued to work, all new customer sessions could not start.
Why it happened:
A change was made to RDC infrastructure, causing device cleaning to fail. This change impacted the permissions devices require to access dependencies during cleaning.
How we fixed it:
The infrastructure configuration change was reverted.
What we are doing to prevent it from happening again:
We are implementing additional guardrails to ensure the infrastructure misconfiguration is not reintroduced in the future.
We are adding additional alerting around device availability, as well as adding resiliency to device cleaning to avoid complete failure based on dependency access.
Resolved
After taking remedial action, Real devices are returning to normal availability in all datacenters. This issue is now resolved
Update
Real devices are showing very limited availability in the US-West-1, US-East-4 and EU-Central-1 datacenters. We are investigating.
Investigating
Real devices are showing very limited availability in the US-West-1 and EU-Central-1 datacenters. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central, US-East), Live Real Device Testing (US-West, EU-Central, US-East), and Native Framework Mobile App Testing (US-West, EU-Central).