Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-May 15 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday May 12th 2023, 10:58 - Monday May 15th 2023, 11:35 UTC
What happened:
Customers experienced intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates.
Why it happened:
One of 4 DNS servers in our data center was upgraded and put back into rotation resulting in DNS resolution failures for hosts querying that host. The root cause was a misconfiguration in the routing table for that node.
How we fixed it:
We fixed the misconfiguration in the routing table.
What we are doing to prevent it from happening again:
We are adding more testing and monitoring for these types of changes.
Resolved
After taking remedial action, WIFI connectivity is back to normal on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. This issue is resolved.
Monitoring
After taking remedial action, we are seeing error rates decreasing in iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. We are currently monitoring.
Investigating
We are experiencing intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).