Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-April-11 Service Incident-2

Incident Report for Sauce Labs

Postmortem

Dates:

Tuesday April 11 2023, 04/11/2023 19:02 UTC - Wednesday April 12 2023, 08:00 UTC

What happened:

Tests running on desktop and virtual mobile devices experienced high wait times and error rates at session start. 

Why it happened:

The data store supporting VM state management reached CPU saturation, causing periodic outages of our VM state management service, each lasting 2-10 minutes in length. As a result, not enough capacity was autoscaled to meet demand.

How we fixed it:

A number of service and infrastructure changes were made to reduce the CPU utilization of the data store.

  • Reducing the number of CPU-heavy queries
  • Splitting some state management workflows to new, separate data stores

What we are doing to prevent it from happening again:

  • Monitoring will be improved to better identify this failure case in the future.
  • We will continue to separate workflows into separate datastores to mitigate against resource saturation and reduce blast radius.
Posted Apr 17, 2023 - 22:38 UTC

Resolved

After taking remedial action, wait times are normal in our US datacenter and all services are fully operational.
Posted Apr 12, 2023 - 00:23 UTC

Update

After taking remedial action, we are seeing improved launch times.
We are currently monitoring.
Posted Apr 12, 2023 - 00:06 UTC

Update

We continue to see elevated wait times in our US datacenter. We continue to investigate.
Posted Apr 11, 2023 - 22:19 UTC

Investigating

We are experiencing elevated wait times in our US datacenter, which may cause tests to be slow to start. We are investigating.
Posted Apr 11, 2023 - 21:48 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Live Browser Testing (US-West), and Live Virtual Mobile Device Testing (US-West).