Sauce Labs Maintenance Windows for Sauce Labs
Sauce Labs has consolidated onto one status page for all data centers.

The old status pages for US-West, US-East and EU-Central are still available to be able to view incident history. Anyone who was subscribed to notifications from one of the old status pages has been migrated onto this consolidated page. If you received an email saying that you had been unsubscribed from notifications, that was from one of the old pages and can be disregarded.

If you would like to select which components that you receive notifications for (e.g. if you are only interested in receiving notifications about incidents affecting Real Device Testing), you can access the subscription management page by clicking "Subscribe To Updates" above and enter the email address you are subscribed with. Additionally there is a "Manage Your Subscription" link on any email you will receive from the status page going forward.
2021-March-11 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday, March 10, 2021, 23:45 - Thursday, March 11, 2021 00:35 UTC

What happened:

The Sauce Labs Real Device Cloud in our EU-Central-1 and US-West-1 data centers was experiencing a high error rate which caused automated and live tests to fail. 

Why it happened:

One of the third-party DNS providers used by Sauce Labs had an outage which caused external DNS lookups to fail. The high error rate caused a slow down on our internal DNS servers which also caused internal lookups to fail during this time frame.

How we fixed it:

After our DNS provider resolved their incident, automated and live test tests on Sauce Labs started to pass again.

What are we doing to prevent it from happening again:

We are migrating a set of internally used DNS names away from our external providers to reduce the load on our internal DNS infrastructure in case of external DNS issues.

Posted Mar 18, 2021 - 19:34 UTC

Resolved
All alerts have resolved and tests are back to their normal state. All services fully operational.
Posted Mar 11, 2021 - 01:04 UTC
Monitoring
We are seeing improvements in DNS resolution and all our services are operating as expected. We continue to monitor.
Posted Mar 11, 2021 - 00:54 UTC
Investigating
We are seeing failures on our eu-central-1 data center as well as on all Legacy RDC tests including the UIs. These are caused by widespread DNS failures detected across the internet. We are investigating and monitoring.
Posted Mar 11, 2021 - 00:41 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central), Billing (EU-Central), Automated Virtual Mobile Device Testing (EU-Central), Legacy RDC REST API (EU-Central), Live Virtual Mobile Device Testing (EU-Central), Live Real Device Testing (EU-Central), Live Browser Testing (EU-Central), Insights (EU-Central), Automated Browser Testing (EU-Central), Sauce Connect (EU-Central), and Legacy RDC UI.