Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-May-05 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday, May 05, 06:09 - 09:59 AM UTC
What happened:
Sauce Connect tunnels for the Legacy RDC platform in our EU data center would not start.
Why it happened:
A server hosting Sauce Connect components was power cycled and restarted with a faulty DNS configuration. A load balancer didn't recognize that the instance was faulty and continued to serve traffic to it.
How we fixed it:
We fixed the faulty DNS configuration which caused the instance to recover.
What we are doing to prevent it from happening again:
We are updating all DNS configurations for the affected cluster to make sure this problem doesn't occur again. In addition, we will improve the health check for this service to make sure we detect faulty instances properly and stop sending traffic to them.
Resolved
This incident has been resolved. All services are operational.
Update
We are continuing to investigate this issue.
Investigating
Legacy RDC Sauce Connect tunnels in our EU Datacenter are not starting. Unified Platform Sauce Connect tunnels are not affected. We are investigating.