Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-October-4 Resolved Service Incident (2)
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday October 4th 2023, 11:00 - 12:01 UTC
What happened:
Customers may have experienced session creation errors when testing with Android Emulators in our EU-Central and US-West data centers.
Why it happened:
The issue was caused by deploying a component responsible for starting sessions in our virtual infrastructure. The code of the new component version contained an error that caused the session processing code to return an HTTP 500 (internal server error) message.
How we fixed it:
When we identified the change causing the issue, we immediately reverted to the previous stable component version. After that, we deployed that component with the latest version, containing a fix.
What we are doing to prevent it from happening again:
- We are adding better static code analysis to the given component, which should prevent errors similar to the one causing this incident.
- We are improving test code coverage of that part of the component.
- We are fine-tuning our alerts so they will alert us much quicker for similar scenarios (too many HTTP 500 returned).
Resolved
Between approximately 11:00 and 12:00 UTC, customers may have experienced session creation errors when testing with Android Emulators in our EU-Central and US-West data centers. This issue has now been resolved.