Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2022-12-13 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday December 13th 2022, 19:05 - 19:27 UTC
What happened:
For approximately 20 minutes, a high number of customers could not start Sauce Connect tunnels in our US-WEST region.
Why it happened:
DNS requests between internal services during this period failed due to high network bandwidth utilization. This caused connection issues during the tunnel initialization process, which caused requests to start tunnels to fail.
How we fixed it:
The high network bandwidth utilization subsided quickly, and DNS requests began to succeed again.
What we are doing to prevent it from happening again:
Our networking team is looking at capacity to see if bottlenecks can be addressed.
Resolved
Between 19:15 UTC and 19:45 UTC, Sauce Connect tunnels were not starting in our US West Data Center. The issue is now resolved. All services are fully operational.