2022-October-25 Resolved Service Incident

Dates:

Tuesday October 25th 2022, 12:32 - 13:33 UTC

What happened:

A small percentage of authentication requests failed causing some users to be unable to log into the Web UI in our EU-Central-1 region. We also observed that some API requests failed during the incident as well. 

Why it happened:

The service that handles authentication requests had one node that was returning errors while attempting to connect to Directory Services. The affected node was using incorrect credentials, which appears to have resulted from a recent deployment of this service. 

How we fixed it:

To re-establish the connection between the Authentication Manager and Directory Services, the team restarted the Authentication Manager nodes. Once this completed, the connection between the services was restored. 

What we are doing to prevent it from happening again:

We have created a new alert that informs teams when this specific condition arises, which will be running continuously but also checked in the post-deployment checks of this service. We are also looking further into how to prevent this password issue in future deployments.