Sauce Labs Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.

2021-Oct-6 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday October 6th 2021, 13:00 - 15:15 UTC

What happened:

Some customers were not able to start Sauce Connect tunnels. Existing running tunnels were not affected.

Why it happened:

Due to an infrastructure upgrade, the tunnel usage cache that we use to calculate tunnel concurrency per user/org contained stale data for several users. It caused the service to drop tunnel start requests that were thought to be in excess of a user allowed tunnel concurrency.

How we fixed it:

We flushed the cache containing stale data and the tunnel concurrency logic started working again.

What we are doing to prevent it from happening again:

We added a cache invalidation logic that runs at intervals to make sure that the cache content is always updated with the correct tunnel usage data.

Posted Oct 14, 2021 - 08:21 UTC

Resolved
We have taken remedial action and tunnels are starting as expected. All services are fully operational.
Posted Oct 06, 2021 - 16:14 UTC
Investigating
Sauce Connect tunnel might fail on startup with "Too many active org tunnels" error on US data center. We are investigating.
Posted Oct 06, 2021 - 15:03 UTC
This incident affected: Sauce Connect (US-West).