2021-March-11 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday, March 10, 2021, 23:45 - Thursday, March 11, 2021 00:35 UTC

What happened:

The Sauce Labs Real Device Cloud in our EU-Central-1 and US-West-1 data centers was experiencing a high error rate which caused automated and live tests to fail.

Why it happened:

One of the third-party DNS providers used by Sauce Labs had an outage which caused external DNS lookups to fail. The high error rate caused a slow down on our internal DNS servers which also caused internal lookups to fail during this time frame.

How we fixed it:

After our DNS provider resolved their incident, automated and live test tests on Sauce Labs started to pass again.

What are we doing to prevent it from happening again:

We are migrating a set of internally used DNS names away from our external providers to reduce the load on our internal DNS infrastructure in case of external DNS issues.

Posted Mar 18, 2021 - 19:34 UTC

Resolved

All alerts have resolved and tests are back to their normal state. All services fully operational.

Posted Mar 11, 2021 - 01:04 UTC

Monitoring

We are seeing improvements in DNS resolution and all our services are operating as expected. We continue to monitor.

Posted Mar 11, 2021 - 00:54 UTC

Investigating

We are seeing failures on our eu-central-1 data center as well as on all Legacy RDC tests including the UIs. These are caused by widespread DNS failures detected across the internet. We are investigating and monitoring.

Posted Mar 11, 2021 - 00:41 UTC

This incident affected: Automated Real Device Testing (US-West, EU-Central), Automated Virtual Mobile Device Testing (EU-Central), Live Browser Testing (EU-Central), Live Virtual Mobile Device Testing (EU-Central), Live Real Device Testing (EU-Central), Insights (EU-Central), Billing (EU-Central), Automated Browser Testing (EU-Central), and Sauce Connect (EU-Central).