Wednesday July 19th 2023, 23:00 UTC - Thursday July 20th 2023, 18:45 UTC
Customers reported automated Android tests failing with the message: "A failure occurred while attempting to upload the video, screenshots, and log files for your test.”
After a deployment, there was a small increase in upload errors that gradually increased overnight. Once it became apparent that there was a problem with the system, we looked closely at all recent changes and determined the deployment referenced earlier introduced an issue with the process that is responsible for uploading test assets to our storage service.
Once the cause was identified, we rolled back to a previously working version.
We have updated the in-team dashboards to more clearly show upload and other errors over a longer period. These dashboards are one of the mechanisms we use to determine if the system is healthy post-deployment.