Maintenance Windows for Sauce Labs
We are seeing issues running automated tests in our Apac-Southeast-1 Data Center, we are currently investigating.
2022-May-20 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday May 20th 2022, 11:50 - 13:15 UTC
What happened:
Sauce Connect tunnels intermittently failed to start in our eu-central-1 data center.
Why it happened:
Unusually high load caused our back-end systems to respond slower than usual.
How we fixed it:
The system restored its normal behavior after the load was decreased.
What we are doing to prevent it from happening again:
- We are planning to improve our back-end system performance in EU-Central to be able to handle this workload
- We will introduce stricter rate-limiting rules to ensure that an individual user cannot affect the system performance
Resolved
Tunnel start in EU Data Center is back to normal. All services are fully operational.
Monitoring
We have taken remedial action and are starting to see improvements. We are monitoring.
Investigating
We are experiencing elevated error rates when starting tunnels in EU Data Center. We are investigating.
This incident affected: Automated Real Device Testing (EU-Central), Native Mobile App Testing (EU-Central), Live Browser Testing (EU-Central), Automated Virtual Mobile Device Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), Sauce Connect (EU-Central), and Automated Browser Testing (EU-Central).