Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-December-15 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday December 14th 2023, 18:36 - Friday December 15th 2023, 14:24 UTC
What happened:
During the incident, tests running on Windows virtual machines did not record a video of the test session.
Why it happened:
A new version of ffmpeg was added to the Windows side disk as part of an unrelated change for Windows. This ffmpeg package had a different directory structure than previous bundles, resulting in the video failing to be recorded.
How we fixed it:
We added temporary logic during the incident to ignore the invalid ffmpeg version on the Windows side disk.
What we are doing to prevent it from happening again:
The team is working on ways to automatically validate video recordings for various OS/Browser versions to detect these issues better.
Resolved
After taking remedial action, test videos are now uploaded successfully for Windows sessions in our US-West and EU-Central Data Centers. All services are fully operational.
Monitoring
We have identified and fixed the root cause. Test result videos are now uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are monitoring.
Investigating
Test videos are not uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central) and Live Browser Testing (US-West, EU-Central).