Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-August-16 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday August 16th 2022, 10:07 - 16:04 UTC
What happened:
A small percentage (< 1%) of users trying to load the test results dashboard in the web UI received an error message and the dashboard was unresponsive.
Why it happened:
The API gateway we use to serve our user authentication services was experiencing intermittent issues that caused calls into those services to fail.
How we fixed it:
We redeployed the pods for the authentication services and then eventually restarted the API gateway which restored service.
What we are doing to prevent it from happening again:
We have added additional monitoring that will detect this specific error condition in the future. We are also expanding error rate alerting across our system and our API Gateway has been a focus of that effort.
Resolved
After taking remedial action, the intermittent errors seen when accessing the Sauce Labs platform UI have been resolved. All services are fully operational.
Update
We are still seeing intermittent errors when accessing the Sauce Labs platform UI on the US-West-1 Data Center. We continue to investigate.
Investigating
We are seeing intermittent errors when accessing the Sauce Labs platform UI on the US-West-1 Data Center. We are investigating.
This incident affected: Sauce Labs Dashboard (US-West).