Dates:

Saturday August 10th 2024, 19:00 - Monday August 12th 2024, 21:13 UTC

What happened:

Following scheduled network upgrades, connectivity between Real Devices and our IPsec VMs was disrupted.  Additionally, a misconfiguration in Sauce Connect caused iOS Real Device Live Testing and app installations to fail.

Why it happened:

During the network upgrade, a community tag was missing from a subset of production routes causing connectivity issues between IPsec VMs and Real Devices.  Additionally, a misconfiguration in Sauce Connect caused iOS Live Testing connectivity issues.

How we fixed it:

To resolve the IPsec issue, we added the missing tags.  To resolve the Sauce Connect with iOS Real Devices issue, we rolled back the Sauce Connect service configuration.

What we are doing to prevent it from happening again:

We’re adding additional end to end testing and alerting for our IPsec and Sauce Connect on iOS products.