Dates:

Wednesday October 19th 2022, 11:45 - 13:10 UTC

What happened:

All iOS devices lost internet connection after a firewall rule was removed.

Why it happened:

iOS devices lost internet connection after a firewall rule was applied that prevented public access to https://app.testobject.com. Our iOS devices have a proxy configured that points to this domain for downloading a “Proxy auto-config” (PAC), and since that failed they lost internet connection.

How we fixed it:

We restored the firewall rule, and connectivity was re-established.

What we are doing to prevent it from happening again:

The team is investigating ways to extract the parts of the system that need internet connectivity. For example, storing the PAC file in publicly accessible cloud storage instead of hosted under https://app.testobject.com.