Dates:

Wednesday August 3rd 2022, 05:10 - 08:17 UTC

What happened:

Customers in our US-EAST-1 region were unable to start new test sessions.

Why it happened:

The process that is responsible for allocating resources to test requests had an abnormal amount of instances running, most of which were not functioning properly. The scheduling services that manages these instances wasn’t cleaning these particular instances correctly. As customers were attempting to start new sessions they were being attached to these instances which prevented the session from starting cleanly.  

How we fixed it:

We manually cleaned up the instances that were not functioning properly. 

What we are doing to prevent it from happening again:

We are addressing the issue in the scheduling service to better detect and clean up instances that aren’t healthy.