Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day
2024-January-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday January 19th 2024, 20:00 - 21:59 UTC
What happened:
An issue with route programming within a service mesh resulted in traffic being dropped to new pods involved in the Sauce Orchestrate request flow.
Why it happened:
A service mesh bug resulted in some routes not being programmed.
How we fixed it:
The control plane of the service mesh was restarted to force a sync event.
What we are doing to prevent it from happening again:
We are improving monitoring for the service mesh and the affected services. Additionally, we are developing a plan for upgrades of the service mesh across the fleet.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Update
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are still investigating.
Investigating
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are currently investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Automated Real Device Testing (US-West).