Maintenance Window
We are experiencing intermittent issues with iOS Simulators and MacOS desktop browser tests in our US-West data center, caused by a third-party service disruption. Our engineering team is actively coordinating with the provider to restore full service.
2023-July-11 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday July 11th 2023, 16:24 - 17:40 UTC
What happened:
We observed an elevated error rate with automated Android tests in the US-WEST region.
Why it happened:
A customer-invoked aggressive retry mechanism caused a set of erroneous tests which skewed our metrics.
How we fixed it:
We worked with the customer to improve their pipeline and the error handling mechanism.
What we are doing to prevent it from happening again:
We’re working on a more restrictive queue mechanism that will throttle such pipelines to not influence our metrics artificially.
Resolved
After taking remedial action, error rates have decreased for Android 13 emulator tests in the US West and EU datacenters. The issue is now resolved.
Investigating
Live and Automated Android 13 emulator tests are seeing elevated error rates in the US West and EU datacenters. We are investigating.
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central) and Live Virtual Mobile Device Testing (US-West, EU-Central).