Sauce Labs Maintenance Windows for Sauce Labs
We are seeing issues running automated tests in our Apac-Southeast-1 Data Center, we are currently investigating.

2022-May-20 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday May 20th 2022, 11:50 - 13:15 UTC

What happened:

Sauce Connect tunnels intermittently failed to start in our eu-central-1 data center.

Why it happened:

Unusually high load caused our back-end systems to respond slower than usual.

How we fixed it:

The system restored its normal behavior after the load was decreased.

What we are doing to prevent it from happening again:

  • We are planning to improve our back-end system performance in EU-Central to be able to handle this workload
  • We will introduce stricter rate-limiting rules to ensure that an individual user cannot affect the system performance
Posted May 31, 2022 - 15:22 UTC

Resolved
Tunnel start in EU Data Center is back to normal. All services are fully operational.
Posted May 20, 2022 - 14:27 UTC
Monitoring
We have taken remedial action and are starting to see improvements. We are monitoring.
Posted May 20, 2022 - 13:56 UTC
Investigating
We are experiencing elevated error rates when starting tunnels in EU Data Center. We are investigating.
Posted May 20, 2022 - 13:37 UTC
This incident affected: Automated Real Device Testing (EU-Central), Native Mobile App Testing (EU-Central), Live Browser Testing (EU-Central), Automated Virtual Mobile Device Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), Sauce Connect (EU-Central), and Automated Browser Testing (EU-Central).