Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2023-May-5 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday May 5th 2023, 10:05 - 11:02 UTC
What happened:
For the duration of the incident, customers experienced slow performance on the Sauce Labs UI, and when executing test requests in all regions.
Why it happened:
A database node became responsive due to a higher than normal number of requests. When this happens, the database failover mechanism typically fires but it failed to do so fully due to a configuration issue.
How we fixed it:
The faulty database node was manually disabled. As a result, all services connected to another node.
What we are doing to prevent it from happening again:
The failover mechanism has been fixed. Next time a similar situation happens, our services will be able to automatically switch to a secondary database node.
Resolved
After taking remedial action, access to the Sauce Labs platform UI has been fully restored, and tests and Sauce Connect tunnels are working across all Data Centers. All services are fully operational.
Investigating
We are currently experiencing slow performance on the Sauce Labs UI and we are seeing errors when starting tests and Sauce Connect tunnels across all Data Centers. We are investigating.
This incident affected: Automated Browser Testing (US-West, US-East (Headless), EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central, US-East), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central, US-East), Sauce Connect (US-West, US-East (Headless), EU-Central), Sauce Labs REST API (US-West, US-East (Headless), EU-Central), Sauce Labs Dashboard (US-West, US-East (Headless), EU-Central), Insights (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).