Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2023-April-29 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Saturday April 29th 2023, 07:03 - 09:06 UTC
What happened:
Customers experienced elevated wait times and error rates for tests running on macOS and iOS simulators for the duration of the incident.
Why it happened:
A third-party automated upgrade erroneously halted services supporting service discovery for one of our data centers. Without service discovery, VMs were not able to be restarted.
How we fixed it:
We restarted the service discovery services, and normal operation resumed.
What we are doing to prevent it from happening again:
We are preventing live upgrades of the service in question, moving it to our regular release workflow.
Resolved
After taking remedial action, MacOS Desktop and iOS Virtual mobile device test wait times and error rates in our US-West-1 Data center have returned to normal levels. This issue is resolved.
Monitoring
After taking remedial action, MacOS Desktop and iOS Virtual mobile device tests wait times and error rates in our US-West-1 Data center are returning to normal levels. We are currently monitoring.
Investigating
MacOS Desktop and iOS Virtual mobile device tests are seeing long wait times and a high rate of errors up in our US-West-1 Data center. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), and Live Virtual Mobile Device Testing (US-West).