2017-May-19 Service Incident
Incident Report for Sauce Labs Inc

Date: May 19, 2017
Time: 9:02am - 9:43am PDT

What Happened:
Some tests did not start or timed out.

Why Did It Happen:
While making an operational enhancement to a server, firewall rules were inadvertently overwritten. This prevented communication between some Sauce Labs services.

How did we fix it:
We fixed the firewall rules.

What we are doing to prevent it from happening again:
- Short term: Updating the appropriate playbook to ensure that firewall rules are corrected when this type of change is made.
- Long term: Changing the networking/NAT architecture to remove a dependency that contributed to the problem.

Posted about 1 month ago. Jun 08, 2017 - 12:32 PDT

Resolved
All systems are fully operational.
Posted 2 months ago. May 19, 2017 - 09:46 PDT
Identified
We identified the root cause and took remedial action. Our service is recovering.
Posted 2 months ago. May 19, 2017 - 09:33 PDT
Investigating
Our system monitoring has detected a problem with PC and Mac clouds. Tests cannot be run.
Posted 2 months ago. May 19, 2017 - 09:29 PDT
This incident affected: Sauce Automated and Sauce Manual.