2022-October-03 Resolved Service Incident

Dates:

Monday October 3rd 2022, 11:56 - 12:20 UTC

What happened:

Real Device testing and the app storage service were unavailable in both the EU and US data centers during the duration of the incident. 

Why it happened:

The service responsible for testing on real devices was not able to decode the JSON web token (JWT) issued by our authentication service, because the deserialization framework used in the JWT library did not accept JWT payloads with unknown fields. 

A deployment of the authentication service introduced a new key to the JWT structure and the RDC and app storage services involved threw an error instead of ignoring the unknown fields. 

How we fixed it:

We performed a rollback of auth-service to the previous version which removed the field from the JWT. 

What we are doing to prevent it from happening again:

We deployed fixes to the impacted services to ignore unknown fields in the JWT instead of throwing an error. We are also looking into better integration testing and API versioning as ways to better manage this in the future.