Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-May-25 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday May 25th 2022, 07:49 - 09:51 UTC.
What happened:
Sauce Connect tunnels intermittently failed to start in our EU-Central-1 data center.
Why it happened:
Unusually high load caused our back-end systems to respond slower than usual.
How we fixed it:
We enabled a more strict rate-limiting to reduce the load caused by a small number of users. The system restored its normal behavior after the load was decreased.
What we are doing to prevent it from happening again:
- We are planning to improve our back-end system performance in EU-Central to be able to handle this workload.
- We are adding more tunnel capacity in our EU datacenter.
Resolved
This incident has been resolved.
Monitoring
Sauce Connect tunnels are now starting normally. We are continuing to monitor the situation.
Identified
The issue has been identified and we are taking steps to recover.
Investigating
Sauce Connect tunnels in our EU-Central data center are intermittently failing to start. We are investigating.
This incident affected: Sauce Connect (EU-Central).