Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2023-July-11 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday July 11th 2023, 16:24 - 17:40 UTC
What happened:
We observed an elevated error rate with automated Android tests in the US-WEST region.
Why it happened:
A customer-invoked aggressive retry mechanism caused a set of erroneous tests which skewed our metrics.
How we fixed it:
We worked with the customer to improve their pipeline and the error handling mechanism.
What we are doing to prevent it from happening again:
We’re working on a more restrictive queue mechanism that will throttle such pipelines to not influence our metrics artificially.
Resolved
After taking remedial action, error rates have decreased for Android 13 emulator tests in the US West and EU datacenters. The issue is now resolved.
Investigating
Live and Automated Android 13 emulator tests are seeing elevated error rates in the US West and EU datacenters. We are investigating.
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central) and Live Virtual Mobile Device Testing (US-West, EU-Central).