2023-January-05 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday January 5th 2022, 00:00 - 02:57 UTC

What happened:

The virtual desktop cloud in our US-West region could not launch new VMs. This led to widespread job failures.

Why it happened:

The system for launching VMs in our virtual desktop cloud became bottlenecked due to a bug that caused unbounded memory usage. This increased memory usage led to cascading failures in connected systems.

How we fixed it:

To remediate the problem, we increased the total amount of memory available to the component causing the bottleneck. We also needed to restart other affected components and reload some data that had dropped due to the memory bottleneck.

What we are doing to prevent it from happening again:

We have fixed the underlying issue that led to the memory usage increase. We are also updating our alerts to get clearer signals in similar scenarios in the future, allowing us to act faster.

Posted Jan 24, 2023 - 10:16 UTC

Resolved

This incident has been resolved. All services are fully operational.

Posted Jan 05, 2023 - 02:58 UTC

Monitoring

We have deployed a fix and are beginning to see improvements. We are monitoring.

Posted Jan 05, 2023 - 02:03 UTC

Investigating

We are seeing elevated error rates and wait times on our Virtual Desktop Cloud in our US West Data Center. We are investigating.

Posted Jan 05, 2023 - 01:00 UTC

This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), and Live Virtual Mobile Device Testing (US-West).