Maintenance Window
We are currently experiencing reduction in available public and private Android devices in the US-West Data Center. We're working to restore full service for all Android users as quickly as possible.
2023-January-11 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday January 11th 2024, 23:17 - Friday January 12th 2024, 01:09 UTC
What happened:
Tests running on Chrome version 120 for any platform and Firefox on Mac experienced failures.
Why it happened:
During a new browser release, a critical update to a dependency for Firefox was not identified due to an irregular and infrequent update cycle for that dependency. Additionally, during the initial rollback, an inconsistent system state affected Chrome.
How we fixed it:
The deployment was reverted. Following the initial revert, an additional deploy was performed to resolve an inconsistent system state.
What we are doing to prevent it from happening again:
We are improving our internal automated testing coverage to include the previously missed component.
Resolved
Between 23:48 UTC and 01:09 UTC, we saw an elevated rate of Infrastructure Errors on our platform impacting browser tests on macOS and Windows in our US West Data Center. We have taken remedial action. All services are fully operational.