2016-July-13 Service Incident
Incident Report for Sauce Labs Inc

What Happened:
Some Mac OSX and iOS tests errored with “Internal Server Error” or timed out due to high wait times.

Why Did It Happen:
A bug was introduced during a deploy that did not occur in our staging environment because the bug is only triggered at significant scale.

How did we fix it:
We reverted the changes made in that deploy.

What we are doing to prevent it from happening again:
- Test cases are being added to better simulate the conditions required to trigger the bug
- A new playbook is being created that would allow us to revert the changes more quickly

Posted 4 months ago. Jul 20, 2017 - 11:51 PDT

Resolved
Our Mac cloud has fully recovered and is operating normally.
Posted 4 months ago. Jul 13, 2017 - 03:50 PDT
Monitoring
Our Mac cloud has recovered. We are monitoring it closely.
Posted 4 months ago. Jul 13, 2017 - 03:32 PDT
Investigating
Our Mac cloud is experiencing high wait times. We’re taking remedial action.
Posted 4 months ago. Jul 13, 2017 - 03:09 PDT
This incident affected: Sauce Automated and Sauce Manual.