Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-July-20 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday July 19th 2023, 23:00 UTC - Thursday July 20th 2023, 18:45 UTC

What happened:

Customers reported automated Android tests failing with the message: "A failure occurred while attempting to upload the video, screenshots, and log files for your test.”

Why it happened:

After a deployment, there was a small increase in upload errors that gradually increased overnight. Once it became apparent that there was a problem with the system, we looked closely at all recent changes and determined the deployment referenced earlier introduced an issue with the process that is responsible for uploading test assets to our storage service. 

How we fixed it:

Once the cause was identified, we rolled back to a previously working version. 

What we are doing to prevent it from happening again:

We have updated the in-team dashboards to more clearly show upload and other errors over a longer period. These dashboards are one of the mechanisms we use to determine if the system is healthy post-deployment.

Posted Aug 04, 2023 - 10:53 UTC

Resolved
After taking remedial action, we are now seeing test assets being uploaded successfully for Android Emulator tests in both US and EU Data Centers. All services are fully operational.
Posted Jul 20, 2023 - 18:50 UTC
Update
We continue to see test asset upload errors for Android Emulator tests in the US and EU Data Centers. We continue to investigate.
Posted Jul 20, 2023 - 18:11 UTC
Investigating
We are currently seeing all Android Emulator tests error with "A failure occurred while attempting to upload the video, screenshots, and log files for your test" in both the US and EU Data Center. We are investigating.
Posted Jul 20, 2023 - 17:19 UTC
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central) and Live Virtual Mobile Device Testing (US-West, EU-Central).