Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2023-May-16 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday May 16th 2023, 04:10 - 06:00 UTC
What happened:
Internal services experienced high response times in short intervals due to an authentication service being hit with a burst of high traffic. This caused Sauce Connect tunnels to shut down, which triggered customer scripts to restart their tunnels automatically.
Why it happened:
Increased traffic from an internal service to an authentication service caused strain on the data store used by the authentication service.
How we fixed it:
The following steps we taken to restore service:
- We fixed a bug in the service making the requests that impacted its effectiveness in rate limiting outbound requests.
- We adjusted the number of retries of the service making requests to the authentication service.
- We adjusted the data store to be able to handle the requests more efficiently.
What we are doing to prevent it from happening again:
We are working with the datastore vendor to make the service more resilient and performant under the conditions experienced during the incident.
Resolved
We have taken corrective action. Sauce Connect tunnels are now starting as normal, and error rates are back to normal levels. This incident is resolved.
Update
We are also seeing Real Device tests using Sauce Connect tunnels intermittently fail. We are still investigating.
Investigating
Sauce Connect tunnels are intermittently not starting and we are seeing an elevated rate of error rate when starting Windows and Android virtual device and desktop browser tests in our EU-Central-1 and US-West-1 data centers. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Sauce Connect (US-West, EU-Central), and Sauce Labs REST API (US-West, EU-Central).