2023-October-23 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Monday October 23rd 2023, 10:21 - 15:12 UTC

What happened:

For the duration of the incident, customers running Live Tests experienced issues with device interactions. Device interaction issues were experienced in all regions and device types for customers with browser resolutions set to anything other than 100%.

Why it happened:

Customers could not interact with devices due to a bug in our Web UI that caused a scaling issue when displaying the canvas on a device. The scaling issue led to failures on the WebSocket that used to handle device interactions. The WebSocket failures ultimately led to the connection closing, leading to all device interactions no longer working.

How we fixed it:

We performed a rollback of our Web UI to the previously known working version.

What we are doing to prevent it from happening again:

After restoring service, we fixed the underlying bug that caused the scaling issues. We are also adding more thorough unit testing for this case.

Posted Nov 20, 2023 - 16:26 UTC

Resolved

After taking remedial action, Android and iOS live tests are now running successfully in our US-West-1 and EU-Central-1 data centers. All services are fully operational.

Posted Oct 23, 2023 - 14:24 UTC

Investigating

We are currently seeing intermittent issues interacting with iOS and Android real devices in live testing, in our US-West-1 and EU-Central-1 data centers. We are investigating.

Posted Oct 23, 2023 - 13:30 UTC

This incident affected: Live Real Device Testing (US-West, EU-Central).