Sauce Labs Maintenance Windows for Sauce Labs
We are currently aware of an ongoing issue impacting the Talkback functionality on our Real Device Cloud. Our team is actively working to resolve this as quickly as possible.
2024-August-12 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Saturday August 10th 2024, 19:00 - Monday August 12th 2024, 21:13 UTC

What happened:

Following scheduled network upgrades, connectivity between Real Devices and our IPsec VMs was disrupted.  Additionally, a misconfiguration in Sauce Connect caused iOS Real Device Live Testing and app installations to fail.

Why it happened:

During the network upgrade, a community tag was missing from a subset of production routes causing connectivity issues between IPsec VMs and Real Devices.  Additionally, a misconfiguration in Sauce Connect caused iOS Live Testing connectivity issues.

How we fixed it:

To resolve the IPsec issue, we added the missing tags.  To resolve the Sauce Connect with iOS Real Devices issue, we rolled back the Sauce Connect service configuration.

What we are doing to prevent it from happening again:

We’re adding additional end to end testing and alerting for our IPsec and Sauce Connect on iOS products.

Posted Sep 23, 2024 - 10:11 UTC

Resolved
After taking remedial action, all services are operating as normal. This incident is resolved.
Posted Aug 12, 2024 - 22:44 UTC
Investigating
We are currently seeing real device iOS installation failures, showing "Unable to Verify App" in the US-West Data Center. We are investigating.
Posted Aug 12, 2024 - 21:13 UTC
This incident affected: Automated Real Device Testing (US-West) and Live Real Device Testing (US-West).