Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2024-February-8 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday February 8th 2024, 12:48 - 15:15 UTC

What happened:

Test execution using Sauce Orchestrate was unavailable.

Why it happened:

An internal queue used by the Live Logging feature of Sauce Orchestrate ran out of space due to an unexpected interaction when configuring a size limit that resulted in reserving more space than was available in the queue cluster.

How we fixed it:

Existing streams were pruned and configurations were updated to use a message limit instead of a size limit.

What we are doing to prevent it from happening again:

We are improving our monitoring of the internal queue service.

Posted Mar 05, 2024 - 16:10 UTC

Resolved

After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Posted Feb 08, 2024 - 16:33 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 08, 2024 - 15:40 UTC

Investigating

We are seeing elevated error rates for tests running via Sauce Orchestrate in the US-West-1 Datacenter. We are investigating.
Posted Feb 08, 2024 - 15:23 UTC
This incident affected: Sauce Orchestrate (US-West).