2023-January-05 Service Incident

We are experiencing an intermittent issue in our US-West-1 datacenter that causes elevated wait times and error rates when starting desktop browser and virtual mobile device tests. Until we address the underlying causes of this issue, we expect this issue to continue to reoccur. We will post incidents here on our status page as each instance of this issue occurs whilst we continue to work to address the underlying cause.

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday January 5th 2022, 00:00 - 02:57 UTC

What happened:

The virtual desktop cloud in our US-West region could not launch new VMs. This led to widespread job failures.

Why it happened:

The system for launching VMs in our virtual desktop cloud became bottlenecked due to a bug that caused unbounded memory usage. This increased memory usage led to cascading failures in connected systems.

How we fixed it:

To remediate the problem, we increased the total amount of memory available to the component causing the bottleneck. We also needed to restart other affected components and reload some data that had dropped due to the memory bottleneck.

What we are doing to prevent it from happening again:

We have fixed the underlying issue that led to the memory usage increase. We are also updating our alerts to get clearer signals in similar scenarios in the future, allowing us to act faster.

Posted Jan 24, 2023 - 10:16 UTC

Resolved

This incident has been resolved. All services are fully operational.

Posted Jan 05, 2023 - 02:58 UTC

Monitoring

We have deployed a fix and are beginning to see improvements. We are monitoring.

Posted Jan 05, 2023 - 02:03 UTC

Investigating

We are seeing elevated error rates and wait times on our Virtual Desktop Cloud in our US West Data Center. We are investigating.

Posted Jan 05, 2023 - 01:00 UTC

This incident affected: Live Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), Automated Browser Testing (US-West), and Automated Virtual Mobile Device Testing (US-West).