Maintenance Windows for Sauce Labs
We are currently aware of an ongoing issue impacting the Talkback functionality on our Real Device Cloud. Our team is actively working to resolve this as quickly as possible.
2024-February-8 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 8th 2024, 12:48 - 15:15 UTC
What happened:
Test execution using Sauce Orchestrate was unavailable.
Why it happened:
An internal queue used by the Live Logging feature of Sauce Orchestrate ran out of space due to an unexpected interaction when configuring a size limit that resulted in reserving more space than was available in the queue cluster.
How we fixed it:
Existing streams were pruned and configurations were updated to use a message limit instead of a size limit.
What we are doing to prevent it from happening again:
We are improving our monitoring of the internal queue service.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are seeing elevated error rates for tests running via Sauce Orchestrate in the US-West-1 Datacenter. We are investigating.
This incident affected: Sauce Orchestrate (US-West).