2018-February 5 Service Incident
Incident Report for Sauce Labs Inc

Date: February 5, 2018
Time: 4:13 pm to 5:35 pm PST

What Happened:
Tests did not start consistently on some Android emulator, iOS simulator and OS X VMs.

Why Did It Happen:
A network configuration error prevented servers in one of the data centers from being able to access essential services.

What did we do to fix it:
We fixed the network configuration manually.

What are we doing to prevent it from happening again:
We permanently set up a whitelist that would prevent this from happening again, and we updated the relevant automated configuration playbooks.

Posted 3 months ago. Feb 12, 2018 - 08:46 PST

Resolved
Tests are running normally again on all of our clouds. All systems are fully operational.
Posted 3 months ago. Feb 05, 2018 - 17:43 PST
Monitoring
Tests have begun to start at normal speeds again. We are monitoring activity on these clouds.
Posted 3 months ago. Feb 05, 2018 - 17:37 PST
Update
Tests are still not starting consistently on our Android emulator, iOS simulator, and OS X clouds.
Posted 3 months ago. Feb 05, 2018 - 17:27 PST
Investigating
Tests on our Android emulator, iOS simulator, and OS X clouds are not starting reliably at the moment. We are investigating.
Posted 3 months ago. Feb 05, 2018 - 16:54 PST
This incident affected: Sauce Automated and Sauce Manual.