2017-May 26 Service Incident
Incident Report for Sauce Labs Inc

Date: May 26, 2017
Time: 12:16-1:23 pm PDT

What Happened:
Tests and Sauce Connect Tunnels could not be started and there were high wait times for VMs.

Why it Happened:
After adding more capacity to the cloud, the DHCP (Dynamic Host Configuration Protocol) service did not restart successfully due to a configuration error. Because VMs for new tests rely on DHCP to obtain an IP address, VMs for new tests were unable to start.

How did we fix it:
We fixed the error in the DHCP configuration file and restarted DHCP.

What are we doing to prevent it from happening again:
- Add checks so that DHCP is not restarted when this configuration error is present.
- Add checks so that if DHCP does not successfully restart, we are notified earlier.

Posted about 1 year ago. Jun 08, 2017 - 12:34 PDT

All services are fully operational.
Posted about 1 year ago. May 26, 2017 - 13:39 PDT
We’ve identified the problem and taken corrective action. Sauce Connect tunnels and tests are starting again. We are monitoring closely.
Posted about 1 year ago. May 26, 2017 - 13:07 PDT
Tests and Sauce Connect tunnels are not starting right now. We are investigating.
Posted about 1 year ago. May 26, 2017 - 12:40 PDT
This incident affected: Manual Testing (Manual VM Testing), Sauce Connect (Sauce Connect VM), Automated VM Testing (Automated PC Testing), REST API (REST API VMs), and Web Interface (Sauce UI, saucelabs.com, Documentation Wiki).