Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2024-February-8 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 8th 2024, 12:48 - 15:15 UTC
What happened:
Test execution using Sauce Orchestrate was unavailable.
Why it happened:
An internal queue used by the Live Logging feature of Sauce Orchestrate ran out of space due to an unexpected interaction when configuring a size limit that resulted in reserving more space than was available in the queue cluster.
How we fixed it:
Existing streams were pruned and configurations were updated to use a message limit instead of a size limit.
What we are doing to prevent it from happening again:
We are improving our monitoring of the internal queue service.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are seeing elevated error rates for tests running via Sauce Orchestrate in the US-West-1 Datacenter. We are investigating.
This incident affected: Sauce Orchestrate (US-West).