Maintenance Window
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day
2023-December-15 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday December 14th 2023, 18:36 - Friday December 15th 2023, 14:24 UTC
What happened:
During the incident, tests running on Windows virtual machines did not record a video of the test session.
Why it happened:
A new version of ffmpeg was added to the Windows side disk as part of an unrelated change for Windows. This ffmpeg package had a different directory structure than previous bundles, resulting in the video failing to be recorded.
How we fixed it:
We added temporary logic during the incident to ignore the invalid ffmpeg version on the Windows side disk.
What we are doing to prevent it from happening again:
The team is working on ways to automatically validate video recordings for various OS/Browser versions to detect these issues better.
Resolved
After taking remedial action, test videos are now uploaded successfully for Windows sessions in our US-West and EU-Central Data Centers. All services are fully operational.
Monitoring
We have identified and fixed the root cause. Test result videos are now uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are monitoring.
Investigating
Test videos are not uploaded for Windows sessions executed on our US-West and EU-Central Data Centers. We are investigating.
This incident affected: Automated Browser Testing (US-West, EU-Central) and Live Browser Testing (US-West, EU-Central).