Date: May 8, 2018
Time: 12:40pm - 1:49pm PDT
Our domains saucelabs.com and ondemand.saucelabs.com did not consistently resolve to the correct IP addresses. Consequently, customers were intermittently unable to run automated tests, use our REST API, start Sauce Connect tunnels, or access our web application.
Although the problem was identified and resolved within an hour, the correction took nearly 72 hours to propagate across the Internet. This is typical for any change made to an authoritative name server. In some cases, corporate DNS caching delayed the update even longer.
Why did it happen:
An engineer inadvertently updated our DNS configuration while investigating SSL certificate warnings.
How did we fix it:
We reverted the incorrect DNS configuration to point back to the correct entries.
What are we doing to prevent it from happening again:
We are moving to a different DNS registrar with better tooling and support. We are also enhancing our change control process, particularly around network changes.