Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-November-28 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Monday November 27 2023, 20:00 -  Tuesday November 28 2023, 15:36 UTC

What happened:

Pixel devices in Europe failed to recover from a specific test scenario, resulting in decreased availability of Pixel devices over time.

Why it happened:

A combination of defects resulted in devices failing to be properly cleaned following a specific customer test scenario involving audio capture.

How we fixed it:

Multiple software defects were fixed and deployed.

What we are doing to prevent it from happening again:

The defects were resolved.  Additionally, new observability work is under investigation to improve detection time for similar availability situations in the future.

Posted Jan 10, 2024 - 08:46 UTC

Resolved
After taking remedial action, all Google Pixel devices are now available in the EU-Central-1 data center.
Posted Nov 28, 2023 - 14:46 UTC
Identified
Public Google Pixel devices continue to be unavailable in the EU-Central-1 data center. We have identified the root cause and are taking remedial action.
Posted Nov 28, 2023 - 11:56 UTC
Investigating
Public Google Pixel devices are currently unavailable in the EU-Central-1 data center. We are investigating.
Posted Nov 28, 2023 - 10:22 UTC
This incident affected: Automated Real Device Testing (EU-Central) and Live Real Device Testing (EU-Central).