2018-March 8 Service Incident
Incident Report for Sauce Labs Inc

Date: March 8, 2018
Time: 3:37 PM - 5:18 PM PST

What Happened:
Wait times for VMs were high and there were periods of tests not starting. This impacted all VM clouds.

Why did it happen:
A service responsible for delivering payloads to our VMs started experiencing issues after a deploy due to a preexisting bug.

What did we do to fix it:
We reverted our payload delivery system to a proven legacy service, which then allowed us to continue booting VM’s normally.

What we are doing to prevent this from happening again:
We’ve fixed the aforementioned bug in the code and are working on deeper refactoring effort, as well as expanded load and functional testing, to eliminate the possibility of this issue reoccurring.

Posted 2 days ago. Mar 16, 2018 - 17:20 PDT

Our service has fully recovered. Tests are running normally and all systems are fully operational.
Posted 10 days ago. Mar 08, 2018 - 17:28 PST
We’ve identified the cause and have taken remedial action. Our cloud is recovering.
Posted 10 days ago. Mar 08, 2018 - 17:11 PST
Tests are not starting currently on our PC, Mac, Emulator, and Simulator clouds. We are investigating.
Posted 10 days ago. Mar 08, 2018 - 16:35 PST
This incident affected: Sauce Automated and Sauce Manual.