Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2021-Oct-6 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday October 6th 2021, 13:00 - 15:15 UTC
What happened:
Some customers were not able to start Sauce Connect tunnels. Existing running tunnels were not affected.
Why it happened:
Due to an infrastructure upgrade, the tunnel usage cache that we use to calculate tunnel concurrency per user/org contained stale data for several users. It caused the service to drop tunnel start requests that were thought to be in excess of a user allowed tunnel concurrency.
How we fixed it:
We flushed the cache containing stale data and the tunnel concurrency logic started working again.
What we are doing to prevent it from happening again:
We added a cache invalidation logic that runs at intervals to make sure that the cache content is always updated with the correct tunnel usage data.
Resolved
We have taken remedial action and tunnels are starting as expected. All services are fully operational.
Investigating
Sauce Connect tunnel might fail on startup with "Too many active org tunnels" error on US data center. We are investigating.
This incident affected: Sauce Connect (US-West).