Maintenance Windows for Sauce Labs
We are experiencing an intermittent issue in our US-West-1 datacenter that causes elevated wait times and error rates when starting desktop browser and virtual mobile device tests. Until we address the underlying causes of this issue, we expect this issue to continue to reoccur. We will post incidents here on our status page as each instance of this issue occurs whilst we continue to work to address the underlying cause.
2023-February-2 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 2nd 2023, 08:15 - 15:27 UTC
What happened:
Appium native app sessions were not starting for specific cases. The error happened for sessions that had:
- For iOS: both “app” and “bundleId” capabilities
- For Android: all of “app”, “appPackage” and “appActivity” capabilities
Why it happened:
After the deployment of a new feature, for the capabilities mentioned above, a bug was introduced in how the combination of those capabilities were processed. The bug caused the processing to be done out of order and caused tests using them to fail to start.
How we fixed it:
The affected service was rolled back to a known safe previous version. A fix for the code was implemented and released afterward.
What we are doing to prevent it from happening again:
Since this is a very sensitive part of the system, extra tests reproducing the issue and documentation were added to the surrounding code. Extra monitoring will be added to detect similar issues in the future.
Resolved
After taking remedial action, all Appium Real Device sessions are starting as expected on both EU-Central-1 and US-West-1 data centers. This issue is now resolved, all services are fully operational.
Investigating
Some Appium sessions are not starting for Android and iOS apps on Real Devices on our US-West-1 and EU-Central-1 data centers. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central).