Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-May 15 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Friday May 12th 2023, 10:58 - Monday May 15th 2023, 11:35 UTC

What happened:

Customers experienced intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates. 

Why it happened:

One of 4 DNS servers in our data center was upgraded and put back into rotation resulting in DNS resolution failures for hosts querying that host. The root cause was a misconfiguration in the routing table for that node.

How we fixed it:

We fixed the misconfiguration in the routing table. 

What we are doing to prevent it from happening again:

We are adding more testing and monitoring for these types of changes.

Posted Jun 02, 2023 - 18:03 UTC

Resolved

After taking remedial action, WIFI connectivity is back to normal on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. This issue is resolved.
Posted May 15, 2023 - 12:59 UTC

Monitoring

After taking remedial action, we are seeing error rates decreasing in iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. We are currently monitoring.
Posted May 15, 2023 - 11:22 UTC

Investigating

We are experiencing intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates. We are investigating.
Posted May 15, 2023 - 10:16 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).