Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-September-28 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday September 28th 2023, 17:05 - 17:36 UTC
What happened:
For the duration of the incident, customers could not start new tests in all regions.
Why it happened:
A change that removed public cloud provider storage interfaces from the core Kubernetes codebase was introduced during a Kubernetes upgrade. We needed to rework large sections of our infrastructure management code to handle this change. In making these changes, we introduced an issue with applying firewall tags that broke key traffic outside the Kubernetes cluster. This was not detected in lower environments.
How we fixed it:
We manually rolled back the configuration that caused the issue.
What we are doing to prevent it from happening again:
We are addressing the underlying issue that caused the broken cluster networking. Once we have that fix we will plan a safe rollout.
Resolved
We have identified and resolved the issue that caused test and tunnel startup failures as well as limited access to our dashboards https://app.saucelabs.com/ in US and EU datacenters.
All services have recovered and are fully operational.
All services have recovered and are fully operational.
Update
We are continuing to investigate this issue.
Investigating
We are currently experiencing test and tunnel startup failures in addition to issues accessing our dashboards https://app.saucelabs.com/ in US and EU datacenters. Requests resolve in 503 Service Unavailable. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central, US-East), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central, US-East), Sauce Labs REST API (US-West, EU-Central), Sauce Labs Dashboard (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).