Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2021-Oct-6 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday October 6th 2021, 13:00 - 15:15 UTC

What happened:

Some customers were not able to start Sauce Connect tunnels. Existing running tunnels were not affected.

Why it happened:

Due to an infrastructure upgrade, the tunnel usage cache that we use to calculate tunnel concurrency per user/org contained stale data for several users. It caused the service to drop tunnel start requests that were thought to be in excess of a user allowed tunnel concurrency.

How we fixed it:

We flushed the cache containing stale data and the tunnel concurrency logic started working again.

What we are doing to prevent it from happening again:

We added a cache invalidation logic that runs at intervals to make sure that the cache content is always updated with the correct tunnel usage data.

Posted Oct 14, 2021 - 08:21 UTC

Resolved

We have taken remedial action and tunnels are starting as expected. All services are fully operational.
Posted Oct 06, 2021 - 16:14 UTC

Investigating

Sauce Connect tunnel might fail on startup with "Too many active org tunnels" error on US data center. We are investigating.
Posted Oct 06, 2021 - 15:03 UTC
This incident affected: Sauce Connect (US-West).