2017-May-4 Service Incident
Incident Report for Sauce Labs Inc

Date: May 4, 2017
Time: 12:05-12:35 pm PDT

What Happened
No automated or manual tests could be started.

Why it Happened
During a past maintenance operation, there was an issue that injected a bug that wasn’t exposed immediately. The bug was encountered during a deploy on May 4 that affected a related area of our Database, causing an issue reading and writing to a key DB schema.

How did we fix it
We fixed the bug and flushed appropriate database tables.

What are we doing to prevent it from happening again
Scanning the database for similar kinds of artifacts and addressing them as needed.

Posted 5 months ago. May 10, 2017 - 08:56 PDT

Resolved
All systems are fully operational.
Posted 5 months ago. May 04, 2017 - 13:14 PDT
Monitoring
Our service is recovering and new tests are starting again. We are closely monitoring.
Posted 5 months ago. May 04, 2017 - 12:50 PDT
Investigating
Automated and manual tests cannot start. We are investigating.
Posted 5 months ago. May 04, 2017 - 12:34 PDT
This incident affected: Sauce Automated, Sauce Manual, Sauce Connect, REST API, Storage REST API, Web Application, saucelabs.com, wiki.saucelabs.com, and Web Application — Archives Page.