Thursday December 2nd 2021, 2:00 - 4:30 UTC
Android devices in the US-West-1 data center failed during regularly recurring health checks and became disabled.
Two services, one responsible for running health checks on our real devices, the other responsible for installing Android applications were installed on the same VM. The installation service has a disk cache that used most of the disk space and impacted the functionality of the service running the health checks. The devices that were checked failed the health check and they were disabled for customer tests.
The services were restarted.
The two services were separated into different VMs. Also the time-to-live of the files in the installation cache was reduced.