Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2024-February-6 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday February 6th 2024, 13:40 - 15:05 UTC
What happened:
Newly-created tests were not being displayed in the Web UI in the Automated and Live Tests sections.
Why it happened:
An updated parsing library failed to parse some messages, resulting in a delay in a Kafka topic.
How we fixed it:
A rollback to the previous version was initiated, at which point prior messages were consumed and the results for affected tests could be viewed in the Web UI.
What we are doing to prevent it from happening again:
We are improving alerting around both the affected service and Kafka.
Resolved
After taking remedial action, test results are being displayed on the live and automated test results pages in all our data centers. All services are fully operational.
Investigating
Live and automated test results are not being displayed on the test results page in all our data centers. We are investigating.
This incident affected: Sauce Labs Dashboard (US-West, EU-Central).