Sauce Labs Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-July-20 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday July 19th 2023, 23:00 UTC - Thursday July 20th 2023, 18:45 UTC

What happened:

Customers reported automated Android tests failing with the message: "A failure occurred while attempting to upload the video, screenshots, and log files for your test.”

Why it happened:

After a deployment, there was a small increase in upload errors that gradually increased overnight. Once it became apparent that there was a problem with the system, we looked closely at all recent changes and determined the deployment referenced earlier introduced an issue with the process that is responsible for uploading test assets to our storage service. 

How we fixed it:

Once the cause was identified, we rolled back to a previously working version. 

What we are doing to prevent it from happening again:

We have updated the in-team dashboards to more clearly show upload and other errors over a longer period. These dashboards are one of the mechanisms we use to determine if the system is healthy post-deployment.

Posted Aug 04, 2023 - 10:53 UTC
Resolved
After taking remedial action, we are now seeing test assets being uploaded successfully for Android Emulator tests in both US and EU Data Centers. All services are fully operational.
Posted Jul 20, 2023 - 18:50 UTC
Update
We continue to see test asset upload errors for Android Emulator tests in the US and EU Data Centers. We continue to investigate.
Posted Jul 20, 2023 - 18:11 UTC
Investigating
We are currently seeing all Android Emulator tests error with "A failure occurred while attempting to upload the video, screenshots, and log files for your test" in both the US and EU Data Center. We are investigating.
Posted Jul 20, 2023 - 17:19 UTC
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central) and Live Virtual Mobile Device Testing (US-West, EU-Central).
← Current Status Powered by Atlassian Statuspage