Sauce Labs Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2023-April-6 Resolved Service Incident
Incident Report for Sauce Labs


Thursday April 6th 2023, 19:16 - 19:33 UTC

What happened:

During the incident, DNS requests made inside one of our EU data centers were failing. There was no direct or discernable impact on customers. 

Why it happened:

A firewall misconfiguration resulted in the loss of the internal DNS network for approximately 17 minutes.

How we fixed it:

The incorrect firewall configuration was cleared. 

What we are doing to prevent it from happening again:

This incident highlights the need for an iptables audit for the internal DNS system with a focus on simplification and standardization. It also uncovers a scenario that the current BGP health checks (which under normal circumstances, would have caused the removal of the misconfigured nodes from the anycast network) cannot detect. Follow up work will be scheduled to resolve both of these issues.

Posted May 03, 2023 - 14:24 UTC

Between 19:16 and 19:33 UTC we experienced DNS related issue impacting Live & Automated tests in our Real device Cloud in the EU-Central-1 datacenter. The underlying issue has been identified and fixed. All services are fully operational.
Posted Apr 06, 2023 - 19:44 UTC