2017-May 26 Service Incident
Incident Report for Sauce Labs Inc

Date: May 26, 2017
Time: 12:16-1:23 pm PDT

What Happened:
Tests and Sauce Connect Tunnels could not be started and there were high wait times for VMs.

Why it Happened:
After adding more capacity to the cloud, the DHCP (Dynamic Host Configuration Protocol) service did not restart successfully due to a configuration error. Because VMs for new tests rely on DHCP to obtain an IP address, VMs for new tests were unable to start.

How did we fix it:
We fixed the error in the DHCP configuration file and restarted DHCP.

What are we doing to prevent it from happening again:
- Add checks so that DHCP is not restarted when this configuration error is present.
- Add checks so that if DHCP does not successfully restart, we are notified earlier.

Posted 6 months ago. Jun 08, 2017 - 12:34 PDT

Resolved
All services are fully operational.
Posted 6 months ago. May 26, 2017 - 13:39 PDT
Identified
We’ve identified the problem and taken corrective action. Sauce Connect tunnels and tests are starting again. We are monitoring closely.
Posted 6 months ago. May 26, 2017 - 13:07 PDT
Investigating
Tests and Sauce Connect tunnels are not starting right now. We are investigating.
Posted 6 months ago. May 26, 2017 - 12:40 PDT
This incident affected: Sauce Automated, Sauce Manual, Sauce Connect, REST API, Storage REST API, Web Application, saucelabs.com, wiki.saucelabs.com, and Web Application — Archives Page.