Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-June-23 Service Incident 2
Incident Report for Sauce Labs
Postmortem

Dates:

Friday June 23, 15:00 - Saturday June 24, 03:00 UTC

What happened:

For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.

Why it happened:

Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.

How we fixed it:

Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.

What we are doing to prevent it from happening again:

This is covered under https://stspg.io/xz9zrrbb6mzs

Posted Jul 11, 2023 - 13:11 UTC

Resolved
After taking remedial action, Sauce Connect tunnels in our US-West-1 data center are starting successfully. All services are fully operational.
Posted Jun 24, 2023 - 03:13 UTC
Investigating
We are seeing intermittent errors with tests using Sauce Connect tunnels in our US-West-1 data center. We are investigating.
Posted Jun 23, 2023 - 15:07 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).