Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2024-February-8 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 8th 2024, 12:48 - 15:15 UTC
What happened:
Test execution using Sauce Orchestrate was unavailable.
Why it happened:
An internal queue used by the Live Logging feature of Sauce Orchestrate ran out of space due to an unexpected interaction when configuring a size limit that resulted in reserving more space than was available in the queue cluster.
How we fixed it:
Existing streams were pruned and configurations were updated to use a message limit instead of a size limit.
What we are doing to prevent it from happening again:
We are improving our monitoring of the internal queue service.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are seeing elevated error rates for tests running via Sauce Orchestrate in the US-West-1 Datacenter. We are investigating.
This incident affected: Sauce Orchestrate (US-West).