2018-Aug-20 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: August 20, 2018
Time: 3:40pm - 7:55pm PDT

*What happened:
*
New tests did not start across all VM platforms.

*Why did it happen:
*
While mitigating a database issue, we encountered a network issue that prevented Operations from completing configuration changes. This caused the system to get into a backed up state, which was further exacerbated by a bug in handling a large backlog of tests.

*How did we fix it:
*
We manually completed the configuration changes and terminated the tests in the backlog.

*What are we doing to prevent it from happening again:
*
We are introducing more process around database changes. The network issue should be resolved by this weekend’s maintenance window. We are also working to make the system more resilient when the load causes a backup in test execution.

Posted 4 months ago. Aug 23, 2018 - 11:35 PDT

Resolved
Service is restored at full capacity. If you experience ongoing problems, please contact us at support.saucelabs.com.
Posted 4 months ago. Aug 20, 2018 - 20:00 PDT
Monitoring
We have applied corrective measures and our service is recovering. We continue to monitor closely.
Posted 4 months ago. Aug 20, 2018 - 19:32 PDT
Update
We are taking remedial actions and monitoring their effects. Most services are still unavailable.
Posted 4 months ago. Aug 20, 2018 - 17:58 PDT
Update
We are continuing to investigate this issue.
Posted 4 months ago. Aug 20, 2018 - 16:57 PDT
Update
We are continuing to investigate this issue.
Posted 4 months ago. Aug 20, 2018 - 16:54 PDT
Investigating
New tests are not starting across all platforms; our Website and REST API are unavailable. We are investigating.
Posted 4 months ago. Aug 20, 2018 - 16:53 PDT
This incident affected: Automated VM Testing (Automated PC Testing, Automated Mac Testing, Automated iOS Simulator Testing, Automated Android Emulator Testing), REST API (REST API VMs, REST API RDC), Manual Testing (Manual VM Testing, Manual RDC Testing), Automated RDC Testing (Automated RDC Testing, EU Data Center, Automated RDC Testing, US Data Center), Web Interface (Sauce UI, Real Device Cloud UI, Analytics, saucelabs.com), and Sauce Connect (Sauce Connect VM, Sauce Connect RDC).