Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-April-6 Resolved Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday April 6th 2023, 19:16 - 19:33 UTC
What happened:
During the incident, DNS requests made inside one of our EU data centers were failing. There was no direct or discernable impact on customers.
Why it happened:
A firewall misconfiguration resulted in the loss of the internal DNS network for approximately 17 minutes.
How we fixed it:
The incorrect firewall configuration was cleared.
What we are doing to prevent it from happening again:
This incident highlights the need for an iptables audit for the internal DNS system with a focus on simplification and standardization. It also uncovers a scenario that the current BGP health checks (which under normal circumstances, would have caused the removal of the misconfigured nodes from the anycast network) cannot detect. Follow up work will be scheduled to resolve both of these issues.
Resolved
Between 19:16 and 19:33 UTC we experienced DNS related issue impacting Live & Automated tests in our Real device Cloud in the EU-Central-1 datacenter. The underlying issue has been identified and fixed. All services are fully operational.