2017-April-18 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: April 18, 2017
Time: 10:30am PDT - 2:47pm PDT

What Happened:
Tests could not be run and Sauce Connect tunnels could not be created.

Why it Happened:
We deployed a change to improve VM boot performance that triggered a problem creating Mac/iOS VMs and Sauce Connect Proxy tunnels on some hosts. The failures resulted in a cycle that subsequently prevented creation of VMs of all OS types.

What we did to fix it:
We reverted the change and patched a core cloud component to stop the cascading failure cycle.

What we are doing to prevent this from happening again:
- Improvements to predictive monitoring.
- Improvements to cloud deploy strategy to allow for more fine grained control, resilience and safer rollbacks.
- Significant investments into performance and reliability of cloud management services

Posted about 1 year ago. Apr 20, 2017 - 15:38 PDT

Resolved
All services are fully operational.
Posted over 1 year ago. Apr 18, 2017 - 15:05 PDT
Monitoring
We have identified the cause, taken remedial actions and our service is recovering. Tests and Sauce Connect tunnels appear to be starting normally. We are closely monitoring.
Posted over 1 year ago. Apr 18, 2017 - 14:47 PDT
Update
Our service still cannot run automated and manual tests, nor start Sauce Connect tunnels. We’re continuing to take remedial action.
Posted over 1 year ago. Apr 18, 2017 - 13:21 PDT
Update
Automated and manual tests still cannot be run. Tunnels start failures are still happening. We continue to investigate.
Posted over 1 year ago. Apr 18, 2017 - 12:27 PDT
Update
Automated and manual tests are not starting. Tunnels are not starting most of the time. We continue to investigate.
Posted over 1 year ago. Apr 18, 2017 - 11:24 PDT
Investigating
Our Mac and PC clouds are experiencing high wait times and Sauce Connect tunnels are not starting. We’re taking remedial action.
Posted over 1 year ago. Apr 18, 2017 - 10:57 PDT
This incident affected: Manual Testing (Manual VM Testing), Sauce Connect (Sauce Connect VM), and Automated VM Testing (Automated PC Testing).