Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-September-12 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Sunday September 12th 2021, 01:34 - 03:58 UTC
What happened:
Some customers experienced slower Sauce Connect tunnel start times in our US-West-1 data center.
Why it happened:
We experienced an increase in requests to start tunnels and failed tunnel connections, leading to the overall system slowing down.
How we fixed it:
After the level of requests to start tunnels returned to normal levels, the number of failed tunneled connections went down, and the system recovered to normal tunnel start times.
What we are doing to prevent it from happening again:
We’re adding more strict rate-limiting rules to ensure that the system cannot be overloaded.
Resolved
Tunnel start time in US is back to normal. All services are fully operational.
Investigating
Tunnels in US are seeing an increase in start time starting at 17:05 PST. We're investigating
This incident affected: Sauce Connect (US-West).