2022-Oct-26 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday, November 26th 2022, 00:32 - 03:03 UTC

What happened:

Customer’s automated tests in our virtual device cloud that were using the latest Chrome version experienced an increased error rate in both the EU and US regions. The message customers received was as follows: Misconfigured -- Unsupported OS/browser/version/device combo: (OS: 'Linux Dev', Browser: 'googlechrome', Version: '107.0.5304.68', …

Why it happened:

The team that manages browser versions usually deploys new browsers in a hidden mode that allows them to test the new browser before they are made generally available to all customers. During a deployment, a pre-release of Chrome browser happened without this hidden tag being applied. Customer tests configured to run on the “latest” version of Chrome began experiencing issues.

How we fixed it:

Once we were alerted to this condition, we removed the pre-release version of Chrome browser from production environments.

What we are doing to prevent it from happening again:

We are reviewing the browser promotion and adding code safeguards in how dev images/browsers are released into our production.

Posted Nov 24, 2022 - 11:57 UTC

Resolved

After taking remedial action, error rates are down on our Desktop tests in EU-Central Data Center. All services are fully operational

Posted Oct 26, 2022 - 03:23 UTC

Investigating

We are seeing an elevated rate of failure on Virtual Desktop Cloud in EU-Central data center. We are investigating.

Posted Oct 26, 2022 - 02:22 UTC

This incident affected: Live Browser Testing (EU-Central) and Automated Browser Testing (EU-Central).