Maintenance Window
We are actively working with our third-party provider to resolve an issue affecting access to the Sauce Labs UI in EU-Central-1 Data Center. This message will remain in place until normal operations resume. Thank you for your patience while we work toward a resolution.
2026-September-28 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday, September 28th 2026, 14:19 UTC - 16:52 UTC.
What happened:
Real device tests in US West hung and failed to output test artifacts due to slow uploads.
Why it happened:
A degraded interconnect link suffered high packet loss from optical errors, but BGP stayed up, preventing automatic failover to healthy paths.
How we fixed it:
Manually disabled the degraded port on the edge router, forcing traffic onto healthy alternate interconnects.
What we are doing to prevent it from happening again:
Automating link draining based on CRC error thresholds, improving protocols for isolating flapping vendor links, and hardening upload resilience.
Resolved
After taking remedial action, Real Device test error rates have returned to normal in the US-West-1 data center. All services are fully operational.
Monitoring
The root cause of the elevated error rates for Real Device tests in the US-West-1 data center has been identified. Remediation has been applied, and we are actively monitoring the situation.
Investigating
We are currently seeing elevated error rates for automated real device tests in US-West-1 data center. We are currently investigating.
This incident affected: Automated Real Device Testing (US-West).