Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-December-15 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Thursday December 14th 2023, 18:36 - Friday December 15th 2023, 14:24 UTC

What happened:

During the incident, tests running on Windows virtual machines did not record a video of the test session.

Why it happened:

A new version of ffmpeg was added to the Windows side disk as part of an unrelated change for Windows. This ffmpeg package had a different directory structure than previous bundles, resulting in the video failing to be recorded. 

How we fixed it:

We added temporary logic during the incident to ignore the invalid ffmpeg version on the Windows side disk.

What we are doing to prevent it from happening again:

The team is working on ways to automatically validate video recordings for various OS/Browser versions to detect these issues better.

Posted Dec 22, 2023 - 11:34 UTC

Resolved
After taking remedial action, test videos are now uploaded successfully for Windows sessions in our US-West and EU-Central Data Centers. All services are fully operational.
Posted Dec 15, 2023 - 15:10 UTC
Monitoring
We have identified and fixed the root cause. Test result videos are now uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are monitoring.
Posted Dec 15, 2023 - 14:44 UTC
Investigating
Test videos are not uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are investigating.
Posted Dec 15, 2023 - 12:39 UTC
This incident affected: Automated Browser Testing (US-West, EU-Central) and Live Browser Testing (US-West, EU-Central).