2021-June-3 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday, June 3 2021, 08:52 - 21:40 UTC

What happened:

Mac browser and iOS Simulator testing was unavailable in our EU-Central-1 data center.

Why it happened:

Two independent events on a provider's fiber network took down two redundant network backhauls that provide connectivity between our Mac infrastructure and the rest of the Sauce Labs EU network. The first event was a planned fiber maintenance. The second event was a fiber outage. While both backhaul circuits are on separate physical paths, these events occurred simultaneously and both circuits were down at the same time.

How we fixed it:

The incident was resolved when one of the circuits was restored. The second circuit was subsequently restored.

What we are doing to prevent it from happening again:

Sauce Labs is actively investigating additional carrier redundancy options at this location to reduce the likelihood of this rare event occurring again.

Posted Jun 10, 2021 - 08:33 UTC

Resolved

Repairs have been completed and we have seen traffic and tests successfully flowing for the past 30 minutes. All services are fully operational.

Posted Jun 03, 2021 - 22:24 UTC

Update

eu-central-1 Automated & Live testing in the Mac Cloud continue to be unavailable due to a fiber cut. Tests may fail to start or fail while running. No Estimated Time to Resolution

Posted Jun 03, 2021 - 21:30 UTC

Update

eu-central-1 Automated & Live testing in the Mac Cloud is unavailable due to a fiber cut. Tests may fail to start or fail while running. No Estimated Time to Resolution

Posted Jun 03, 2021 - 16:06 UTC

Identified

We are working with a third-party Network Provider to restore connectivity to the eu-central-1 Mac Cloud

Posted Jun 03, 2021 - 11:43 UTC

Investigating

Our eu-central-1 Mac Cloud (Mac Desktop and iOS Simulator testing) is unavailable. Tests may fail to start or fail while running.

Posted Jun 03, 2021 - 09:19 UTC

This incident affected: Automated Virtual Mobile Device Testing (EU-Central), Live Browser Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), and Automated Browser Testing (EU-Central).