2023-September-28 Service Incident 2

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday September 28th 2023, 21:10 UTC - Friday September 29th 2023, 00:43 UTC

What happened:

Users attempting to use live test sessions in the Sauce Labs UI were either given an error or a blank screen for the duration of the incident.

Why it happened:

When attempting to insert a non-serializable value into our memory store, an error was thrown, preventing actions such as starting a live session. This error began occurring after a middleware was added to our memory store that attempted to serialize any payload, which, upon failing to do so, would throw an error preventing that action from being committed.

How we fixed it:

We rolled back to the previous version of live testing to restore service.

What we are doing to prevent it from happening again:

We are adding a blocking stage to our pipeline to check for these issues by running live sessions for both real devices and web sessions.

Posted Oct 10, 2023 - 13:29 UTC

Resolved

We have identified and resolved the issue that caused Live sessions on Real Device Cloud to fail to start in US and EU datacenters. All services are fully operational.

Posted Sep 29, 2023 - 00:45 UTC

Update

We continue investigating the issue with running Real Device app and browser tests in US West and EU datacenters. Cross-browser sessions are running in both US and EU datacenters.

Posted Sep 28, 2023 - 23:48 UTC

Investigating

We are seeing an issue with running Live Sessions in US West and EU datacenters. Real device and Web browser Live tests fail to start. We are investigating.

Posted Sep 28, 2023 - 23:00 UTC

This incident affected: Live Browser Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).