Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-Oct-6 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday October 6th 2021, 13:00 - 15:15 UTC
What happened:
Some customers were not able to start Sauce Connect tunnels. Existing running tunnels were not affected.
Why it happened:
Due to an infrastructure upgrade, the tunnel usage cache that we use to calculate tunnel concurrency per user/org contained stale data for several users. It caused the service to drop tunnel start requests that were thought to be in excess of a user allowed tunnel concurrency.
How we fixed it:
We flushed the cache containing stale data and the tunnel concurrency logic started working again.
What we are doing to prevent it from happening again:
We added a cache invalidation logic that runs at intervals to make sure that the cache content is always updated with the correct tunnel usage data.
Resolved
We have taken remedial action and tunnels are starting as expected. All services are fully operational.
Investigating
Sauce Connect tunnel might fail on startup with "Too many active org tunnels" error on US data center. We are investigating.
This incident affected: Sauce Connect (US-West).