2018-October-7 Resolved Issue
Incident Report for Sauce Labs Inc
Postmortem

Date: October 7, 2018
Time: 12:51 pm - 1:09 pm PDT

*What happened:
*
Tests could not be run, Sauce Connect tunnels could not be started, and the web application and REST API were unavailable.

*Why it happened:
*
A memory module in a database server failed and caused the database to run out of memory.

*How we fixed it:
*
We failed over to the secondary database server.

*What we are doing to prevent it from happening again:
*
We plan to make the database failover process more automated.

Posted 3 days ago. Oct 19, 2018 - 11:41 PDT

Resolved
On Sunday October 7, 2018, at about 1 pm PST, most parts of our service (running tests, Sauce Connect, the Web UI) stopped functioning. The root cause, a primary database server that ran out of memory, was identified in less than ten minutes. We failed over to a secondary database server and our service returned to full normal operation within the next five minutes. The total period of customer impact was about fifteen minutes. This will count as downtime against our service. Some customers may have had to restart tunnels after the event or seen tests fail during this period. Currently, all services are fully operational.
Posted 14 days ago. Oct 08, 2018 - 13:44 PDT