Friday June 23, 15:00 - Saturday June 24, 03:00 UTC
For the duration of the incident, we saw intermittent periods where ~25% of tests in our US-West data center did not start.
Requests to open and close firewall access between the test and tunnel vm were timing out. This caused a build-up in unprocessed requests, further slowing processing time, and causing requests for new tests to time out.
Solutions to resolve the incident included: manually processing close access requests and using alternate access between the tunnel and test VM where possible.
This is covered under https://stspg.io/xz9zrrbb6mzs