Dates:

Monday April 24th 2023, 15:35 - 16:40 UTC

What happened:

During the incident, the percentage of real devices in an error state increased to about 20%. 

• The majority impacted devices were running Android 12 and 13
• Android 11 and below remained in a healthy state
• Private devices were also impacted
• The devices only entered an errored state after running a test

Why it happened:

Android toolkit had to be compiled with the latest stable SDK to be able to run on Android 14 beta 1. With the latest SDK, it’s no longer possible to call the hidden system API through reflection with the current configuration. Due to this change, the Android toolkit failed to clean the device Locale, which is a mandatory cleaning step. If that step fails, then the device immediately goes into an error state. 

How we fixed it:

We rolled back the Android toolkit to an older version compiled with an older SDK.

What we are doing to prevent it from happening again:

We are looking into ways to handle device cleaning better in these scenarios, for example, in some cases cleaning device locale isn’t always needed. We are also improving how we test and detect this type of issue throughout our development lifecycle.