Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2022-12-13 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday December 13th 2022, 19:05 - 19:27 UTC
What happened:
For approximately 20 minutes, a high number of customers could not start Sauce Connect tunnels in our US-WEST region.
Why it happened:
DNS requests between internal services during this period failed due to high network bandwidth utilization. This caused connection issues during the tunnel initialization process, which caused requests to start tunnels to fail.
How we fixed it:
The high network bandwidth utilization subsided quickly, and DNS requests began to succeed again.
What we are doing to prevent it from happening again:
Our networking team is looking at capacity to see if bottlenecks can be addressed.
Resolved
Between 19:15 UTC and 19:45 UTC, Sauce Connect tunnels were not starting in our US West Data Center. The issue is now resolved. All services are fully operational.