Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-May 15 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday May 12th 2023, 10:58 - Monday May 15th 2023, 11:35 UTC
What happened:
Customers experienced intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates.
Why it happened:
One of 4 DNS servers in our data center was upgraded and put back into rotation resulting in DNS resolution failures for hosts querying that host. The root cause was a misconfiguration in the routing table for that node.
How we fixed it:
We fixed the misconfiguration in the routing table.
What we are doing to prevent it from happening again:
We are adding more testing and monitoring for these types of changes.
Resolved
After taking remedial action, WIFI connectivity is back to normal on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. This issue is resolved.
Monitoring
After taking remedial action, we are seeing error rates decreasing in iOS Real devices in our US-West-1 & EU-Central-1 Data Centers. We are currently monitoring.
Investigating
We are experiencing intermittent WiFi connection failures on iOS Real devices in our US-West-1 & EU-Central-1 Data Centers, leading to increased error rates. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).