Maintenance Windows for Sauce Labs
We are experiencing an intermittent issue in our US-West-1 datacenter that causes elevated wait times and error rates when starting desktop browser and virtual mobile device tests. Until we address the underlying causes of this issue, we expect this issue to continue to reoccur. We will post incidents here on our status page as each instance of this issue occurs whilst we continue to work to address the underlying cause.
2022-12-13 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday December 13th 2022, 19:05 - 19:27 UTC
What happened:
For approximately 20 minutes, a high number of customers could not start Sauce Connect tunnels in our US-WEST region.
Why it happened:
DNS requests between internal services during this period failed due to high network bandwidth utilization. This caused connection issues during the tunnel initialization process, which caused requests to start tunnels to fail.
How we fixed it:
The high network bandwidth utilization subsided quickly, and DNS requests began to succeed again.
What we are doing to prevent it from happening again:
Our networking team is looking at capacity to see if bottlenecks can be addressed.
Resolved
Between 19:15 UTC and 19:45 UTC, Sauce Connect tunnels were not starting in our US West Data Center. The issue is now resolved. All services are fully operational.