Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-July-11 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Tuesday July 11th 2023, 16:24 - 17:40 UTC

What happened:

We observed an elevated error rate with automated Android tests in the US-WEST region.

Why it happened:

A customer-invoked aggressive retry mechanism caused a set of erroneous tests which skewed our metrics.

How we fixed it:

We worked with the customer to improve their pipeline and the error handling mechanism.

What we are doing to prevent it from happening again:

We’re working on a more restrictive queue mechanism that will throttle such pipelines to not influence our metrics artificially.

Posted Jul 17, 2023 - 15:11 UTC

Resolved

After taking remedial action, error rates have decreased for Android 13 emulator tests in the US West and EU datacenters. The issue is now resolved.
Posted Jul 11, 2023 - 17:41 UTC

Investigating

Live and Automated Android 13 emulator tests are seeing elevated error rates in the US West and EU datacenters. We are investigating.
Posted Jul 11, 2023 - 16:39 UTC
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central) and Live Virtual Mobile Device Testing (US-West, EU-Central).