2023-May 16 Service Incident 3

Incident Report for Sauce Labs

Postmortem

Dates:

Sunday May 14th 2023, 07:44 - Tuesday May 16th 2023, 15:15 UTC

What happened:

A change made to the default direct domains configuration caused a failure for customers who had specified their own direct domains when starting a new Sauce Connect tunnel. This resulted in either a tunnel that wasn’t working or a tunnel that would fail to start with a cryptic error message.

Why it happened:

A change in the default direct domains configuration conflicted with existing customer settings, causing their Sauce Connect tunnels to fail and shut down after startup.

How we fixed it:

We reverted the change to the previous configuration.

What we are doing to prevent it from happening again:

We deployed a longer-term fix for this issue that validates the combination of customer domains and default domains that are put in place. We are also adding more testing and monitoring to the proxy service that was at the center of this incident.

Posted Jun 02, 2023 - 18:12 UTC

Resolved

After taking remedial action, Sauce Connect tunnels have returned to normal operation in our US-West-1 and EU-Central-1 Data Centers. This incident is resolved.

Posted May 16, 2023 - 15:33 UTC

Identified

We are seeing errors for some customers when starting Sauce Connect tunnels in our US-West-1 & EU-Central-1 Data Centers. We are investigating.

Posted May 16, 2023 - 14:42 UTC

This incident affected: Automated Browser Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central, US-East), Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central, US-East), Sauce Connect (US-West, EU-Central), and Sauce Labs REST API (US-West, EU-Central).