Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2023-June-23 Service Incident 2

Incident Report for Sauce Labs

Postmortem

Dates:

Friday June 23, 15:00 - Saturday June 24, 03:00 UTC

What happened:

For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.

Why it happened:

Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.

How we fixed it:

Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.

What we are doing to prevent it from happening again:

This is covered under https://stspg.io/xz9zrrbb6mzs

Posted Jul 11, 2023 - 13:11 UTC

Resolved

After taking remedial action, Sauce Connect tunnels in our US-West-1 data center are starting successfully. All services are fully operational.
Posted Jun 24, 2023 - 03:13 UTC

Investigating

We are seeing intermittent errors with tests using Sauce Connect tunnels in our US-West-1 data center. We are investigating.
Posted Jun 23, 2023 - 15:07 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).