Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-June-23 Service Incident 2

Incident Report for Sauce Labs

Postmortem

Dates:

Friday June 23, 15:00 - Saturday June 24, 03:00 UTC

What happened:

For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.

Why it happened:

Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.

How we fixed it:

Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.

What we are doing to prevent it from happening again:

This is covered under https://stspg.io/xz9zrrbb6mzs

Posted Jul 11, 2023 - 13:11 UTC

Resolved

After taking remedial action, Sauce Connect tunnels in our US-West-1 data center are starting successfully. All services are fully operational.
Posted Jun 24, 2023 - 03:13 UTC

Investigating

We are seeing intermittent errors with tests using Sauce Connect tunnels in our US-West-1 data center. We are investigating.
Posted Jun 23, 2023 - 15:07 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).