Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2024-January-19 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday January 19th 2024, 20:00 - 21:59 UTC

What happened:

An issue with route programming within a service mesh resulted in traffic being dropped to new pods involved in the Sauce Orchestrate request flow.

Why it happened:

A service mesh bug resulted in some routes not being programmed.

How we fixed it:

The control plane of the service mesh was restarted to force a sync event.

What we are doing to prevent it from happening again:

We are improving monitoring for the service mesh and the affected services. Additionally, we are developing a plan for upgrades of the service mesh across the fleet.

Posted Jan 29, 2024 - 17:39 UTC

Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Posted Jan 19, 2024 - 12:09 UTC
Update
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are still investigating.
Posted Jan 19, 2024 - 11:57 UTC
Investigating
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are currently investigating.
Posted Jan 19, 2024 - 11:15 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Automated Real Device Testing (US-West).