Dates:

Thursday May 19th 2022, 01:50 - 3:10 UTC

What happened:

Our Real Device Cloud in the EU was not available to start any new tests for automated and manual tests. No devices were visible and accessible for running tests.

Why it happened:

A bad change to the database collection that includes the configuration for real mobile devices was conducted. Our services received an error when they were trying to read the collection to get information about which devices were connected to the real device cloud in Europe.

Change to the device configuration of a single device was done directly in the database. 

The tool that was used to change the database was faulty. Necessary diligence and checks to prevent a change like that to be allowed to happen were not in place.

How we fixed it:

The faulty document was removed. 

What we are doing to prevent it from happening again:

We are limiting access to the database further, and adding additional checks. 

We will also improve our internal tooling that allows us to do changes to device configurations in a safer way.We are adding additional alerts that trigger an immediate response for similar scenarios.