Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-October-19 Resolved Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday October 19th 2022, 11:45 - 13:10 UTC

What happened:

All iOS devices lost internet connection after a firewall rule was removed.

Why it happened:

iOS devices lost internet connection after a firewall rule was applied that prevented public access to https://app.testobject.com. Our iOS devices have a proxy configured that points to this domain for downloading a “Proxy auto-config” (PAC), and since that failed they lost internet connection.

How we fixed it:

We restored the firewall rule, and connectivity was re-established.

What we are doing to prevent it from happening again:

The team is investigating ways to extract the parts of the system that need internet connectivity. For example, storing the PAC file in publicly accessible cloud storage instead of hosted under https://app.testobject.com.

Posted Nov 08, 2022 - 14:31 UTC

Resolved
Between 12:45 UTC and 13:15 UTC We experienced decreased availability and internet connectivity in iOS real devices in both US-West-1 & EU-Central-1 datacenters. This issue is now resolved, all services are fully operational.
Posted Oct 19, 2022 - 13:23 UTC