Thursday January 5th 2022, 00:00 - 02:57 UTC
The virtual desktop cloud in our US-West region could not launch new VMs. This led to widespread job failures.
The system for launching VMs in our virtual desktop cloud became bottlenecked due to a bug that caused unbounded memory usage. This increased memory usage led to cascading failures in connected systems.
To remediate the problem, we increased the total amount of memory available to the component causing the bottleneck. We also needed to restart other affected components and reload some data that had dropped due to the memory bottleneck.
We have fixed the underlying issue that led to the memory usage increase. We are also updating our alerts to get clearer signals in similar scenarios in the future, allowing us to act faster.