2018-May-8 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: May 8, 2018
Time: 12:40pm - 1:49pm PDT

What happened:
Our domains saucelabs.com and ondemand.saucelabs.com did not consistently resolve to the correct IP addresses. Consequently, customers were intermittently unable to run automated tests, use our REST API, start Sauce Connect tunnels, or access our web application.

Although the problem was identified and resolved within an hour, the correction took nearly 72 hours to propagate across the Internet. This is typical for any change made to an authoritative name server. In some cases, corporate DNS caching delayed the update even longer.

Why did it happen:
An engineer inadvertently updated our DNS configuration while investigating SSL certificate warnings.

How did we fix it:
We reverted the incorrect DNS configuration to point back to the correct entries.

What are we doing to prevent it from happening again:
We are moving to a different DNS registrar with better tooling and support. We are also enhancing our change control process, particularly around network changes.

Posted 5 months ago. May 15, 2018 - 15:33 PDT

Resolved
DNS update is in place and propagation is expected to be ongoing for next 24 hours. All services are fully operational.
Posted 6 months ago. May 08, 2018 - 14:37 PDT
Monitoring
We have identified the problem and corrected it. DNS update will take some time to propagate and take full effect. We’ll continue to monitor the situation.
Posted 6 months ago. May 08, 2018 - 14:14 PDT
Investigating
We are experiencing intermittent DNS issue affecting domains saucelabs.com and ondemand.saucelabs.com. Intermittently, reaching REST API, Sauce Labs web application, and running automated tests may fail. We are investigating.
Posted 6 months ago. May 08, 2018 - 14:03 PDT
This incident affected: REST API (REST API VMs), Web Interface (Sauce UI), and Automated VM Testing (Automated PC Testing).