Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-Nov-30 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Thursday, December 1st 2022, 02:08 - 04:05 UTC

What happened:

Automated tests in our virtual device cloud that were configured to use the latest Chrome browser version experienced an increased session start time or elevated error rate in both the EU and US regions. 

In these scenarios, users saw the following error messages:

  • Infrastructure Error -- The Sauce VMs failed to start the browser or device.
  • Session did not start. User might have disconnected. 

Why it happened:

A new Chrome browser release process left some stale images/VMs unable to handle the latest release of the Chrome browser. Customer tests configured to run on the “latest” version of Chrome began experiencing issues.

How we fixed it:

Once we were alerted to this condition, we decided to revert the latest Chrome browser release from production environments.

What we are doing to prevent it from happening again:

We are reviewing the browser promotion and adding code safeguards in how images/browsers are released into our production.

Posted Dec 20, 2022 - 13:47 UTC

Resolved
We have taken corrective action and Mac browser tests are running as expected in all datacenters. All services are fully operational.
Posted Dec 01, 2022 - 04:38 UTC
Investigating
We are seeing Mac desktop tests failing to start in EU Central and US West datacenters. We are investigating.
Posted Dec 01, 2022 - 03:50 UTC
This incident affected: Automated Browser Testing (US-West, EU-Central) and Live Browser Testing (US-West, EU-Central).