Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-June-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday June 19th 2023, 15:19 - 15:56 UTC
What happened:
An infrastructure change caused devices in all RDC data centers to falsely report as busy. While ongoing sessions continued to work, all new customer sessions could not start.
Why it happened:
A change was made to RDC infrastructure, causing device cleaning to fail. This change impacted the permissions devices require to access dependencies during cleaning.
How we fixed it:
The infrastructure configuration change was reverted.
What we are doing to prevent it from happening again:
We are implementing additional guardrails to ensure the infrastructure misconfiguration is not reintroduced in the future.
We are adding additional alerting around device availability, as well as adding resiliency to device cleaning to avoid complete failure based on dependency access.
Resolved
After taking remedial action, Real devices are returning to normal availability in all datacenters. This issue is now resolved
Update
Real devices are showing very limited availability in the US-West-1, US-East-4 and EU-Central-1 datacenters. We are investigating.
Investigating
Real devices are showing very limited availability in the US-West-1 and EU-Central-1 datacenters. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central, US-East), Live Real Device Testing (US-West, EU-Central, US-East), and Native Framework Mobile App Testing (US-West, EU-Central).