2018-March 8 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: March 8, 2018
Time: 3:37 PM - 5:18 PM PST

What Happened:
Wait times for VMs were high and there were periods of tests not starting. This impacted all VM clouds.

Why did it happen:
A service responsible for delivering payloads to our VMs started experiencing issues after a deploy due to a preexisting bug.

What did we do to fix it:
We reverted our payload delivery system to a proven legacy service, which then allowed us to continue booting VM’s normally.

What we are doing to prevent this from happening again:
We’ve fixed the aforementioned bug in the code and are working on deeper refactoring effort, as well as expanded load and functional testing, to eliminate the possibility of this issue reoccurring.

Posted 6 months ago. Mar 16, 2018 - 17:20 PDT

Resolved
Our service has fully recovered. Tests are running normally and all systems are fully operational.
Posted 7 months ago. Mar 08, 2018 - 17:28 PST
Monitoring
We’ve identified the cause and have taken remedial action. Our cloud is recovering.
Posted 7 months ago. Mar 08, 2018 - 17:11 PST
Investigating
Tests are not starting currently on our PC, Mac, Emulator, and Simulator clouds. We are investigating.
Posted 7 months ago. Mar 08, 2018 - 16:35 PST
This incident affected: Manual Testing (Manual VM Testing) and Automated VM Testing (Automated PC Testing).