Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-May-20 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday May 20th 2022, 11:50 - 13:15 UTC
What happened:
Sauce Connect tunnels intermittently failed to start in our eu-central-1 data center.
Why it happened:
Unusually high load caused our back-end systems to respond slower than usual.
How we fixed it:
The system restored its normal behavior after the load was decreased.
What we are doing to prevent it from happening again:
- We are planning to improve our back-end system performance in EU-Central to be able to handle this workload
- We will introduce stricter rate-limiting rules to ensure that an individual user cannot affect the system performance
Resolved
Tunnel start in EU Data Center is back to normal. All services are fully operational.
Monitoring
We have taken remedial action and are starting to see improvements. We are monitoring.
Investigating
We are experiencing elevated error rates when starting tunnels in EU Data Center. We are investigating.
This incident affected: Automated Virtual Mobile Device Testing (EU-Central), Automated Real Device Testing (EU-Central), Live Browser Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), Native Framework Mobile App Testing (EU-Central), Automated Browser Testing (EU-Central), and Sauce Connect (EU-Central).