Sauce Labs Maintenance Windows for Sauce Labs
Our EU Central data center (RDC) will undergo unplanned but necessary power maintenance on June 19th, 2025, between 07:00 and 18:00 CEST (05:00–16:00 UTC). During this maintenance window, devices may experience brief service interruptions lasting up to 20–30 minutes. We are taking every measure to minimize impact and ensure full service restoration as quickly as possible. We appreciate your understanding and apologize for any inconvenience this may cause.

2023-April-6 Service Incident 2

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday April 6th 2023, 21:52 - 23:44 UTC

What happened:

The service responsible for starting tunnels and assigning jobs to tunnels failed due to increased memory usage. This affected all jobs in the US West data center as they needed to be checked for tunnel access.

Why it happened:

There was a bottleneck in our tunnel services that caused increased memory usage under heavy load.

How we fixed it:

We modified the tunnel services to distribute requests better, and to reduce memory consumption when those requests were being made.

What we are doing to prevent it from happening again:

The initial condition causing the increased memory usage on the tunnel services has been resolved, and we will improve our monitoring to alert earlier on high memory usage conditions.

Posted May 03, 2023 - 14:08 UTC

Resolved

This incident has been resolved. All services are fully operational.
Posted Apr 07, 2023 - 01:05 UTC

Monitoring

We have taken remedial action and are beginning to see improvements starting new tunnels and running tests using existing tunnels. We are monitoring.
Posted Apr 06, 2023 - 23:59 UTC

Update

We are seeing an increased rate of test failure in our US West DataCenter affecting both tunneled and non-tunneled tests. We continue to investigate.
Posted Apr 06, 2023 - 22:52 UTC

Investigating

We are experiencing issues with starting tunnels and running tests using existing tunnels in our US West Data Center. We are investigating.
Posted Apr 06, 2023 - 22:28 UTC
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), API Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).