2022-May-25 Service Incident 2

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday May 25th 2022, 07:49 - 09:51 UTC.

What happened:

Sauce Connect tunnels intermittently failed to start in our EU-Central-1 data center.

Why it happened:

Unusually high load caused our back-end systems to respond slower than usual.

How we fixed it:

We enabled a more strict rate-limiting to reduce the load caused by a small number of users. The system restored its normal behavior after the load was decreased.

What we are doing to prevent it from happening again:

We are planning to improve our back-end system performance in EU-Central to be able to handle this workload.
We are adding more tunnel capacity in our EU datacenter.

Posted Jun 09, 2022 - 16:15 UTC

Resolved

This incident has been resolved.

Posted May 25, 2022 - 10:36 UTC

Monitoring

Sauce Connect tunnels are now starting normally. We are continuing to monitor the situation.

Posted May 25, 2022 - 10:11 UTC

Identified

The issue has been identified and we are taking steps to recover.

Posted May 25, 2022 - 09:48 UTC

Investigating

Sauce Connect tunnels in our EU-Central data center are intermittently failing to start. We are investigating.

Posted May 25, 2022 - 08:13 UTC

This incident affected: Sauce Connect (EU-Central).