Monday June 19th 2023, 15:19 - 15:56 UTC
An infrastructure change caused devices in all RDC data centers to falsely report as busy. While ongoing sessions continued to work, all new customer sessions could not start.
A change was made to RDC infrastructure, causing device cleaning to fail. This change impacted the permissions devices require to access dependencies during cleaning.
The infrastructure configuration change was reverted.
We are implementing additional guardrails to ensure the infrastructure misconfiguration is not reintroduced in the future.
We are adding additional alerting around device availability, as well as adding resiliency to device cleaning to avoid complete failure based on dependency access.