2021-Dec-02 Service Incident

Dates:

Thursday December 2nd 2021, 2:00 - 4:30 UTC

What happened:

Android devices in the US-West-1 data center failed during regularly recurring health checks and became disabled.

Why it happened:

Two services, one responsible for running health checks on our real devices, the other responsible for installing Android applications were installed on the same VM. The installation service has a disk cache that used most of the disk space and impacted the functionality of the service running the health checks. The devices that were checked failed the health check and they were disabled for customer tests.

How we fixed it:

The services were restarted. 

What we are doing to prevent it from happening again:

The two services were separated into different VMs. Also the time-to-live of the files in the installation cache was reduced.