2022-October-27 Resolved Service Incident

Dates:

Friday October 27th 2022, 09:15 - 09:45 UTC

What happened:

We observed higher than normal response times from our user service, which impacted all functionality that depends on it. The largest impact was elevated wait times, errors when attempting to start virtual device tests, and errors in the dashboard and test results in our US West data center. 

Why it happened:

Over the past year, we have migrated most of the services responsible for test orchestration to Google Cloud. One of these services is our user service, which relies on a data store with a component still configured to run in our on-premise data center. 

While cleaning up resources for our on-premise deployments, we removed a configuration that pointed the user service to an on-premise component that increased latency when accessing Forgerock DS. 

How we fixed it:

Since this component was no longer needed, we removed the configuration and the on-premise resources. Then we restarted the service, and the latency went away as everything was running in Google Cloud. 

What we are doing to prevent it from happening again:

Now that all components are cleaned up, we are safe from this reoccurring.