Date: May 19, 2017
Time: 9:02am - 9:43am PDT
Some tests did not start or timed out.
Why Did It Happen:
While making an operational enhancement to a server, firewall rules were inadvertently overwritten. This prevented communication between some Sauce Labs services.
How did we fix it:
We fixed the firewall rules.
What we are doing to prevent it from happening again:
- Short term: Updating the appropriate playbook to ensure that firewall rules are corrected when this type of change is made.
- Long term: Changing the networking/NAT architecture to remove a dependency that contributed to the problem.