Wednesday March 6th 2024, 12:43 - 19:00 UTC
Test artifact uploads and downloads intermittently failed or were slower to complete, resulting in issues displaying test results.
A subsystem involved in artifact uploads would not properly recover from some failures, resulting in leaked connections and connection pool exhaustion. Once fully exhausted, uploads were no longer possible.
We identified the bug in the code and deployed a fix.
We have fixed the specific bug and implemented monitoring for similar symptoms.