Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2023-April-11 Service Incident-2
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday April 11 2023, 04/11/2023 19:02 UTC - Wednesday April 12 2023, 08:00 UTC
What happened:
Tests running on desktop and virtual mobile devices experienced high wait times and error rates at session start.
Why it happened:
The data store supporting VM state management reached CPU saturation, causing periodic outages of our VM state management service, each lasting 2-10 minutes in length. As a result, not enough capacity was autoscaled to meet demand.
How we fixed it:
A number of service and infrastructure changes were made to reduce the CPU utilization of the data store.
- Reducing the number of CPU-heavy queries
- Splitting some state management workflows to new, separate data stores
What we are doing to prevent it from happening again:
- Monitoring will be improved to better identify this failure case in the future.
- We will continue to separate workflows into separate datastores to mitigate against resource saturation and reduce blast radius.
Resolved
After taking remedial action, wait times are normal in our US datacenter and all services are fully operational.
Update
After taking remedial action, we are seeing improved launch times.
We are currently monitoring.
We are currently monitoring.
Update
We continue to see elevated wait times in our US datacenter. We continue to investigate.
Investigating
We are experiencing elevated wait times in our US datacenter, which may cause tests to be slow to start. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), and Live Virtual Mobile Device Testing (US-West).