2017-June-20 Service Incident
Incident Report for Sauce Labs Inc

Date of Incident: June 20, 2017
Time of Incident: 12:05 pm PST to 1:55 pm PST

What Happened: REST API calls began to return errors in the 500s at a high rate. The error rate started at 15% and climbed to 99% over the next hour. Services dependent on making successful REST API calls, such as our Web Application and the Sauce Connect client, failed at a similar rate.

Why It Happened: Our database experienced an unusually high volume of traffic, the origin of which is still being investigated. Resource usage increased to critically high levels. As query response times increased, API calls and downstream services like our Web Application and Sauce Connect, which are dependent on database queries, started to timeout and fail.

What We Did To Fix It: We terminated all database processes and released all the SQL handles on the buffer pool. This stopped the contention for memory and allowed query response times to return to normal.

What we are doing to prevent this from happening again: We're performing an exhaustive analysis of our existing queries with a focus on eliminating potential problem sources and improving their overall safety and efficiency.

Posted 5 months ago. Jun 26, 2017 - 16:30 PDT

Resolved
Our services have fully recovered and are operational. All users of Sauce Connect should restart their tunnels now.
Posted 5 months ago. Jun 20, 2017 - 14:08 PDT
Monitoring
Our services are recovering. We are monitoring them closely.
Posted 5 months ago. Jun 20, 2017 - 13:56 PDT
Update
Our service continues to have difficulties loading pages of the web interface, starting tunnels, and responding to requests using our REST API. We’re taking remedial action.
Posted 5 months ago. Jun 20, 2017 - 13:40 PDT
Investigating
We are having intermittent failures with our web interface, starting tunnels, and our REST API. We are investigating.
Posted 5 months ago. Jun 20, 2017 - 13:00 PDT
This incident affected: Sauce Manual, Sauce Connect, REST API, Storage REST API, Web Application, and Web Application — Archives Page.