Some Mac OSX and iOS tests errored with “Internal Server Error” or timed out due to high wait times.
Why Did It Happen:
A bug was introduced during a deploy that did not occur in our staging environment because the bug is only triggered at significant scale.
How did we fix it:
We reverted the changes made in that deploy.
What we are doing to prevent it from happening again:
- Test cases are being added to better simulate the conditions required to trigger the bug
- A new playbook is being created that would allow us to revert the changes more quickly