Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-October-23 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Monday October 23rd 2023, 10:21 - 15:12 UTC

What happened:

For the duration of the incident, customers running Live Tests experienced issues with device interactions. Device interaction issues were experienced in all regions and device types for customers with browser resolutions set to anything other than 100%. 

Why it happened:

Customers could not interact with devices due to a bug in our Web UI that caused a scaling issue when displaying the canvas on a device. The scaling issue led to failures on the WebSocket that used to handle device interactions. The WebSocket failures ultimately led to the connection closing, leading to all device interactions no longer working. 

How we fixed it:

We performed a rollback of our Web UI to the previously known working version.

What we are doing to prevent it from happening again:

After restoring service, we fixed the underlying bug that caused the scaling issues. We are also adding more thorough unit testing for this case.

Posted Nov 20, 2023 - 16:26 UTC
Resolved
After taking remedial action, Android and iOS live tests are now running successfully in our US-West-1 and EU-Central-1 data centers. All services are fully operational.
Posted Oct 23, 2023 - 14:24 UTC
Investigating
We are currently seeing intermittent issues interacting with iOS and Android real devices in live testing, in our US-West-1 and EU-Central-1 data centers. We are investigating.
Posted Oct 23, 2023 - 13:30 UTC
This incident affected: Live Real Device Testing (US-West, EU-Central).
← Current Status Powered by Atlassian Statuspage