Dates:

Wednesday September 6th 2023, 18:35 UTC - Thursday September 7th 19:05 UTC

What happened:

A subset of customers were unable to run tests with specific uploaded apps in the EU region intermittently for the duration of the incident. 

Why it happened:

During migrating users from regional to global teams, several organizations in the EU had their users moved to the default team, meaning they were unable to run tests using apps uploaded by another team. Various scripts were created to prepare and execute this migration, one of which was meant to determine a user’s region. This script had a bug in it and misconfigured 55 organizations.

How we fixed it:

We were able to restore the user configuration to restore service, and then we addressed the bug in the script and completed the migration. 

What we are doing to prevent it from happening again:

We have addressed bugs in the migration pipeline and the scripts it leverages for future runs.