Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-September-27 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday September 27th 2022, 17:59 - 20:08 UTC

What happened:

During the duration of the incident, users were unable to start Real Device tests on devices below iOS 13.2. If a user attempted to start a session the device would appear to be busy and eventually would time out. 

Why it happened:

A new feature called WebRTC was deployed for iOS devices which required us to install a service application on our real iOS based devices. This service app was to be installed during our device cleaning process.This new feature was intended to go out for only devices running iOS 13 and higher. This was intended to be controlled by setting an iOS TARGET version in Xcode but due to a bug in that control it went out to all devices. 

When the service application was installed on devices running iOS versions lower than 13.2 it failed. During device cleaning if any step should fail the device is marked “dirty” and isn’t available for tests.

How we fixed it:

We tried to rollback the rdc-pool version which contains this WebRTC service app.We ran into two issues:

  • We were not able to rollback, because this component is configured to prevent multiple deployments within 24 hours.
  • We had to SSH into every single individual instance of this component, stop the queued deployment image, kill the service, and enable the new rolled back version.

What we are doing to prevent it from happening again:

We identified two areas to improve:

  • Enable a more expedient deployment mechanism for this component.
  • Fixing the bug that prevented us from targeting specific iOS versions.
Posted Nov 08, 2022 - 13:39 UTC

Resolved
After taking remedial action, all iOS real devices are now available in both the US and EU data centers. All services are fully operational.
Posted Sep 27, 2022 - 20:12 UTC
Update
We are still seeing that iOS real devices with versions 13.2 and below are unavailable in both the US and EU data centers. We continue to investigate.
Posted Sep 27, 2022 - 18:52 UTC
Investigating
We are seeing iOS devices with versions 13.2 and below are unavailable in both the US and EU data centers. We are investigating.
Posted Sep 27, 2022 - 18:13 UTC
This incident affected: Live Real Device Testing (US-West, EU-Central), Native Framework Mobile App Testing (US-West, EU-Central), and Automated Real Device Testing (US-West, EU-Central).