Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-Dec-07 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday December 7th 2021, 7:30 - 8:30 UTC
What happened:
Visual Testing services, including the Visual Testing UI, were unavailable and tests failed to start.
Why it happened:
Unanticipated side effects of some database maintenance caused database connections to drop.
How we fixed it:
Services were able to reconnect once the database maintenance was complete.
What we are doing to prevent it from happening again:
We are working to understand the possible risk of future maintenance and will perform this work during scheduled maintenance windows if necessary.
Resolved
This incident has been resolved.
Monitoring
Visual services are recovering and new builds are now running correctly. Builds that were started whilst services were impacted will timeout. We are continuing to monitor the situation.
Investigating
Our Visual Testing service is unavailable. We are investigating.
This incident affected: Visual Testing (Visual Testing Hub, Visual Testing Infrastructure, Visual Testing REST API, Legacy Visual Testing (Screener) UI).