2018-September-28 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: September 28, 2018
Time: 10:05 am - 10:35 am PDT

*What happened:
*
Automated tests could not be run. The web application and REST API were unavailable.

*Why it happened:
*
Statistic collection on the database caused contention during a period of high customer activity.

*How we fixed it:
*
Once statistic collection was finished, the application was able to operate normally.

*What we are doing to prevent it from happening again:
*
We have tuned the statistic collection parameters to greatly reduce the overhead of statistic collection and are investigating scheduling it during off peak hours.

Posted 2 months ago. Oct 05, 2018 - 11:38 PDT

Resolved
Our systems have recovered completely. All services are fully operational.
Posted 2 months ago. Sep 28, 2018 - 12:16 PDT
Update
Customers may see elevated wait times on new PC Automated and Manual tests. Our website is back to normal.
Posted 2 months ago. Sep 28, 2018 - 11:47 PDT
Monitoring
Website and tests are recovering. We'll continue to monitor the situation.
Posted 2 months ago. Sep 28, 2018 - 10:53 PDT
Update
Website and tests are recovering. We'll continue to monitor the situation.
Posted 2 months ago. Sep 28, 2018 - 10:46 PDT
Investigating
All our services are currently unavailable. We are investigating.
Posted 2 months ago. Sep 28, 2018 - 10:33 PDT
This incident affected: Automated VM Testing (Automated PC Testing, Automated Mac Testing, Automated iOS Simulator Testing, Automated Android Emulator Testing), Manual Testing (Manual VM Testing), and Web Interface (Sauce UI, saucelabs.com).