Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-August-25 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday August 25, 2021 00:30 05:30 UTC
What happened:
Customers running tests on our US-West and EU-Central data centers were unable to start Sauce Connect tunnels. Customers running tests on existing tunnels were not impacted.
Why it happened:
A faulty script was used to clean up old tunnel images. This script inadvertently deleted empty folders on all images, including active images. Some of these empty folders were required for tunnel startup.
How we fixed it:
We deployed a new tunnel image. New tunnels are created with this image, while existing tunnels continue to run.
What are we doing to prevent it from happening again:
We are reviewing and replacing our process for deploying and updating tunnel images, and will ensure all use cases are documented and tested.
Resolved
We have taken remedial action and tunnels are now starting as expected on all DC's. All services are fully operational.
Identified
We believe we've found what is causing tunnels to not start. We are working on a fix and expect to deploy a fix soon.
Update
New Sauce Connect tunnel fails to start on our US and EU data center. We continue to investigate.
Investigating
New Sauce Connect tunnels will not start successfully on our US and EU data center. Existing tunnels are not affected. We are investigating.
This incident affected: Sauce Connect (US-West, EU-Central).