2023-November-1 Service Incident

Dates:

Friday November 1st 2023, 10:15 - 18:00 UTC

What happened:

An individual customer was continuously running a large number of misconfigured tests, impacting the overall error rate in the US-West region and ultimately triggering our alerting system. Though the error rate alerting threshold was surpassed, no other customer tests were impacted.

Why it happened:

Due to the individual customer’s misconfiguration, all tests they ran failed to start. Our test validation service did not properly identify the misconfiguration, causing VMs to be allocated to them. The large number of failing tests tripped our alerting system, though only the individual customer was impacted.

How we fixed it:

We worked with the individual customer to stop and reconfigure their tests.

What we are doing to prevent it from happening again:

We are enhancing our test configuration validation to better inform customers of these issues before a VM is allocated to a test that will ultimately fail. We are also improving our visibility into test errors, to better identify and monitor individual customer issues versus overall system issues.