Sauce Labs Maintenance Windows for Sauce Labs
We are seeing issues running automated tests in our Apac-Southeast-1 Data Center, we are currently investigating.

2022-May-25 Service Incident 2
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday May 25th 2022, 07:49 - 09:51 UTC.

What happened:

Sauce Connect tunnels intermittently failed to start in our EU-Central-1 data center.

Why it happened:

Unusually high load caused our back-end systems to respond slower than usual.

How we fixed it:

We enabled a more strict rate-limiting to reduce the load caused by a small number of users. The system restored its normal behavior after the load was decreased.

What we are doing to prevent it from happening again:

  • We are planning to improve our back-end system performance in EU-Central to be able to handle this workload.
  • We are adding more tunnel capacity in our EU datacenter.
Posted Jun 09, 2022 - 16:15 UTC

Resolved
This incident has been resolved.
Posted May 25, 2022 - 10:36 UTC
Monitoring
Sauce Connect tunnels are now starting normally. We are continuing to monitor the situation.
Posted May 25, 2022 - 10:11 UTC
Identified
The issue has been identified and we are taking steps to recover.
Posted May 25, 2022 - 09:48 UTC
Investigating
Sauce Connect tunnels in our EU-Central data center are intermittently failing to start. We are investigating.
Posted May 25, 2022 - 08:13 UTC
This incident affected: Sauce Connect (EU-Central).