Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day
2024-February-8 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 8th 2024, 12:48 - 15:15 UTC
What happened:
Test execution using Sauce Orchestrate was unavailable.
Why it happened:
An internal queue used by the Live Logging feature of Sauce Orchestrate ran out of space due to an unexpected interaction when configuring a size limit that resulted in reserving more space than was available in the queue cluster.
How we fixed it:
Existing streams were pruned and configurations were updated to use a message limit instead of a size limit.
What we are doing to prevent it from happening again:
We are improving our monitoring of the internal queue service.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are seeing elevated error rates for tests running via Sauce Orchestrate in the US-West-1 Datacenter. We are investigating.
This incident affected: US-West.