2018-February 28 Service Incident
Incident Report for Sauce Labs Inc
Postmortem

Date: February 28, 2018
Time: 7:00am - 11:15am PST

What Happened:
Users experienced a 20-40% error rate when accessing our service's web application or using its API.

Why did it happen:
An early implementation of a new feature was allowed to remain generally available alongside a more performative version of the same feature. An increase in usage of the less performant version of this feature caused an unexpected amount of resource pressure to occur which degraded the feature, as well as other related services.

What did we do to fix it:
We disabled the early implementation of the feature and directed user traffic to the more performative solution.

What we are doing to prevent this from happening again:
In addition to migrating user traffic to the more performative solution, we’ll be implementing more proactive feature migration processes and deepening our existing application performance monitoring to ensure a faster response to any future issues.

Posted 7 months ago. Mar 08, 2018 - 15:01 PST

Resolved
All affected services fully recovered and are fully operational.
Posted 7 months ago. Feb 28, 2018 - 11:28 PST
Monitoring
We identified the issue and took remedial actions. Our response times are back to normal and affected services function as expected. We are monitoring the situation.
Posted 7 months ago. Feb 28, 2018 - 11:17 PST
Update
SauceLabs dashboard, and manual testing interface response is slow or responds with error. REST API and consequently Sauce Connect tunnels are also affected. We continue to investigate.
Posted 7 months ago. Feb 28, 2018 - 09:58 PST
Update
SauceLabs dashboard, and manual testing interface response is slow or responds with error. REST API and consequently Sauce Connect tunnels are also affected. We continue to investigate.
Posted 7 months ago. Feb 28, 2018 - 09:28 PST
Investigating
SauceLabs dashboard and manual testing interface response is slow or responds with error. We are investigating.
Posted 7 months ago. Feb 28, 2018 - 09:04 PST
This incident affected: Sauce Connect (Sauce Connect VM), Web Interface (Sauce UI), Manual Testing (Manual VM Testing), and REST API (REST API VMs).