2024-August-12 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Saturday August 10th 2024, 19:00 - Monday August 12th 2024, 21:13 UTC

What happened:

Following scheduled network upgrades, connectivity between Real Devices and our IPsec VMs was disrupted. Additionally, a misconfiguration in Sauce Connect caused iOS Real Device Live Testing and app installations to fail.

Why it happened:

During the network upgrade, a community tag was missing from a subset of production routes causing connectivity issues between IPsec VMs and Real Devices. Additionally, a misconfiguration in Sauce Connect caused iOS Live Testing connectivity issues.

How we fixed it:

To resolve the IPsec issue, we added the missing tags. To resolve the Sauce Connect with iOS Real Devices issue, we rolled back the Sauce Connect service configuration.

What we are doing to prevent it from happening again:

We’re adding additional end to end testing and alerting for our IPsec and Sauce Connect on iOS products.

Posted Sep 23, 2024 - 10:11 UTC

Resolved

After taking remedial action, all services are operating as normal. This incident is resolved.

Posted Aug 12, 2024 - 22:44 UTC

Investigating

We are currently seeing real device iOS installation failures, showing "Unable to Verify App" in the US-West Data Center. We are investigating.

Posted Aug 12, 2024 - 21:13 UTC

This incident affected: Automated Real Device Testing (US-West) and Live Real Device Testing (US-West).