Maintenance Window
We are currently experiencing reduction in available public and private Android devices in the US-West Data Center. We're working to restore full service for all Android users as quickly as possible.
2022-12-13 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday December 13th 2022, 19:05 - 19:27 UTC
What happened:
For approximately 20 minutes, a high number of customers could not start Sauce Connect tunnels in our US-WEST region.
Why it happened:
DNS requests between internal services during this period failed due to high network bandwidth utilization. This caused connection issues during the tunnel initialization process, which caused requests to start tunnels to fail.
How we fixed it:
The high network bandwidth utilization subsided quickly, and DNS requests began to succeed again.
What we are doing to prevent it from happening again:
Our networking team is looking at capacity to see if bottlenecks can be addressed.
Resolved
Between 19:15 UTC and 19:45 UTC, Sauce Connect tunnels were not starting in our US West Data Center. The issue is now resolved. All services are fully operational.