Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-June-23 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday June 23, 15:00 - Saturday June 24, 03:00 UTC
What happened:
For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.
Why it happened:
Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.
How we fixed it:
Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.
What we are doing to prevent it from happening again:
This is covered under https://stspg.io/xz9zrrbb6mzs
Resolved
After taking remedial action, Sauce Connect tunnels in our US-West-1 data center are starting successfully. All services are fully operational.
Investigating
We are seeing intermittent errors with tests using Sauce Connect tunnels in our US-West-1 data center. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).