Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-February-2 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday February 2nd 2023, 08:15 - 15:27 UTC
What happened:
Appium native app sessions were not starting for specific cases. The error happened for sessions that had:
- For iOS: both “app” and “bundleId” capabilities
- For Android: all of “app”, “appPackage” and “appActivity” capabilities
Why it happened:
After the deployment of a new feature, for the capabilities mentioned above, a bug was introduced in how the combination of those capabilities were processed. The bug caused the processing to be done out of order and caused tests using them to fail to start.
How we fixed it:
The affected service was rolled back to a known safe previous version. A fix for the code was implemented and released afterward.
What we are doing to prevent it from happening again:
Since this is a very sensitive part of the system, extra tests reproducing the issue and documentation were added to the surrounding code. Extra monitoring will be added to detect similar issues in the future.
Resolved
After taking remedial action, all Appium Real Device sessions are starting as expected on both EU-Central-1 and US-West-1 data centers. This issue is now resolved, all services are fully operational.
Investigating
Some Appium sessions are not starting for Android and iOS apps on Real Devices on our US-West-1 and EU-Central-1 data centers. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central).