2018-November-29 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: November 29, 2018
Time: 9:39am - 10:34am, 12:28pm - 12:34pm

What happened:
The REST API and the web application (app.saucelabs.com) were slow to respond.

Why it happened:
Unusual load from customers caused strain on our API’s ability to serve requests, both to our API consumers and the web application.

How we fixed it:
The overall load on the system decreased, allowing the system to resume normal operations.

What we are doing to prevent it from happening again:
A number of performance improvements are being implemented which will allow the API to function well under an excessive amount of load. Additionally, better monitoring will be implemented to allow us to continue to proactively tune performance as load continues to ramp up.

Posted 2 months ago. Dec 12, 2018 - 13:11 PST

Resolved
We've made changes to our WebUI that allow for better communication and load balancing with our REST API. Performance has improved and the intermittent issues appear to have been resolved. All services are fully operational.
Posted 3 months ago. Nov 29, 2018 - 17:23 PST
Update
Our Web UI and REST API have been functioning normally, aside from one ten minute period of slowness around 12:30pm PST. We are continuing to look for the root cause.
Posted 3 months ago. Nov 29, 2018 - 14:15 PST
Update
Our Web UI and REST API are functioning normally at the moment, but the root cause is still unknown and periods of slowness may return. We are continuing to investigate.
Posted 3 months ago. Nov 29, 2018 - 11:23 PST
Investigating
Our Web UI and REST API are intermittently unavailable or responding slowly. We are investigating.
Posted 3 months ago. Nov 29, 2018 - 10:31 PST
This incident affected: REST API (REST API VMs) and Web Interface (Sauce UI, saucelabs.com).