Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-September-20 Resolved Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday September 20th 2023, 09:39 - 15:38 UTC

What happened:

Sauce Connect users could not load their websites behind Sauce Connect tunnels until they closed a pop-up informing them, "Your connection is not private”.

Why it happened:

Windows virtual machines did not import the certificates needed to run tunnels properly due to a change in how they were stored on the virtual machine. A startup script references several locations to load certificates from, and not all of them were updated to reflect the new location on the virtual machine. 

How we fixed it:

The startup script was updated to reflect the new certificate locations. 

What we are doing to prevent it from happening again:

We reviewed the tunnel startup code across all platforms to ensure no other areas referenced the old locations. We also added unit tests that check whether there is a mismatch between where the certificates are stored and their references in the startup script.

Posted Sep 28, 2023 - 15:06 UTC

Resolved
Between 10:39 UTC and 15:38 UTC our desktop tests that use Sauce Connect certificate resign feature experienced failures due to invalid certificates on US-West-1 and EU-Central-1 datacenters . We have identified the issue and taken remedial action.
Posted Sep 20, 2023 - 17:59 UTC