2017-May-4 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: May 4, 2017
Time: 12:05-12:35 pm PDT

What Happened
No automated or manual tests could be started.

Why it Happened
During a past maintenance operation, there was an issue that injected a bug that wasn’t exposed immediately. The bug was encountered during a deploy on May 4 that affected a related area of our Database, causing an issue reading and writing to a key DB schema.

How did we fix it
We fixed the bug and flushed appropriate database tables.

What are we doing to prevent it from happening again
Scanning the database for similar kinds of artifacts and addressing them as needed.

Posted about 1 year ago. May 10, 2017 - 08:56 PDT

Resolved
All systems are fully operational.
Posted about 1 year ago. May 04, 2017 - 13:14 PDT
Monitoring
Our service is recovering and new tests are starting again. We are closely monitoring.
Posted about 1 year ago. May 04, 2017 - 12:50 PDT
Investigating
Automated and manual tests cannot start. We are investigating.
Posted about 1 year ago. May 04, 2017 - 12:34 PDT
This incident affected: Manual Testing (Manual VM Testing), Sauce Connect (Sauce Connect VM), Automated VM Testing (Automated PC Testing), REST API (REST API VMs), and Web Interface (Sauce UI, saucelabs.com, Documentation Wiki).