Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2023-June-23 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday June 23, 15:00 - Saturday June 24, 03:00 UTC
What happened:
For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.
Why it happened:
Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.
How we fixed it:
Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.
What we are doing to prevent it from happening again:
This is covered under https://stspg.io/xz9zrrbb6mzs
Resolved
After taking remedial action, Sauce Connect tunnels in our US-West-1 data center are starting successfully. All services are fully operational.
Investigating
We are seeing intermittent errors with tests using Sauce Connect tunnels in our US-West-1 data center. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).