Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-September-28 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday September 28th 2023, 17:05 - 17:36 UTC
What happened:
For the duration of the incident, customers could not start new tests in all regions.
Why it happened:
A change that removed public cloud provider storage interfaces from the core Kubernetes codebase was introduced during a Kubernetes upgrade. We needed to rework large sections of our infrastructure management code to handle this change. In making these changes, we introduced an issue with applying firewall tags that broke key traffic outside the Kubernetes cluster. This was not detected in lower environments.
How we fixed it:
We manually rolled back the configuration that caused the issue.
What we are doing to prevent it from happening again:
We are addressing the underlying issue that caused the broken cluster networking. Once we have that fix we will plan a safe rollout.
Resolved
We have identified and resolved the issue that caused test and tunnel startup failures as well as limited access to our dashboards https://app.saucelabs.com/ in US and EU datacenters.
All services have recovered and are fully operational.
All services have recovered and are fully operational.
Update
We are continuing to investigate this issue.
Investigating
We are currently experiencing test and tunnel startup failures in addition to issues accessing our dashboards https://app.saucelabs.com/ in US and EU datacenters. Requests resolve in 503 Service Unavailable. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central, US-East), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central, US-East), Sauce Labs REST API (US-West, EU-Central), Sauce Labs Dashboard (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).