2017-April-18 Service Incident
Incident Report for Sauce Labs Inc

Date: April 18, 2017
Time: 10:30am PDT - 2:47pm PDT

What Happened:
Tests could not be run and Sauce Connect tunnels could not be created.

Why it Happened:
We deployed a change to improve VM boot performance that triggered a problem creating Mac/iOS VMs and Sauce Connect Proxy tunnels on some hosts. The failures resulted in a cycle that subsequently prevented creation of VMs of all OS types.

What we did to fix it:
We reverted the change and patched a core cloud component to stop the cascading failure cycle.

What we are doing to prevent this from happening again:
- Improvements to predictive monitoring.
- Improvements to cloud deploy strategy to allow for more fine grained control, resilience and safer rollbacks.
- Significant investments into performance and reliability of cloud management services

Posted 5 months ago. Apr 20, 2017 - 15:38 PDT

Resolved
All services are fully operational.
Posted 5 months ago. Apr 18, 2017 - 15:05 PDT
Monitoring
We have identified the cause, taken remedial actions and our service is recovering. Tests and Sauce Connect tunnels appear to be starting normally. We are closely monitoring.
Posted 5 months ago. Apr 18, 2017 - 14:47 PDT
Update
Our service still cannot run automated and manual tests, nor start Sauce Connect tunnels. We’re continuing to take remedial action.
Posted 5 months ago. Apr 18, 2017 - 13:21 PDT
Update
Automated and manual tests still cannot be run. Tunnels start failures are still happening. We continue to investigate.
Posted 5 months ago. Apr 18, 2017 - 12:27 PDT
Update
Automated and manual tests are not starting. Tunnels are not starting most of the time. We continue to investigate.
Posted 5 months ago. Apr 18, 2017 - 11:24 PDT
Investigating
Our Mac and PC clouds are experiencing high wait times and Sauce Connect tunnels are not starting. We’re taking remedial action.
Posted 5 months ago. Apr 18, 2017 - 10:57 PDT
This incident affected: Sauce Automated, Sauce Manual, and Sauce Connect.