Maintenance Windows for Sauce Labs
We are currently aware of an ongoing issue impacting the Talkback functionality on our Real Device Cloud. Our team is actively working to resolve this as quickly as possible.
2023-December-15 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday December 14th 2023, 18:36 - Friday December 15th 2023, 14:24 UTC
What happened:
During the incident, tests running on Windows virtual machines did not record a video of the test session.
Why it happened:
A new version of ffmpeg was added to the Windows side disk as part of an unrelated change for Windows. This ffmpeg package had a different directory structure than previous bundles, resulting in the video failing to be recorded.
How we fixed it:
We added temporary logic during the incident to ignore the invalid ffmpeg version on the Windows side disk.
What we are doing to prevent it from happening again:
The team is working on ways to automatically validate video recordings for various OS/Browser versions to detect these issues better.
Resolved
After taking remedial action, test videos are now uploaded successfully for Windows sessions in our US-West and EU-Central Data Centers. All services are fully operational.
Monitoring
We have identified and fixed the root cause. Test result videos are now uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are monitoring.
Investigating
Test videos are not uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central) and Live Browser Testing (US-West, EU-Central).