2021-March-23 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates

Tuesday, March 23, 11:07 - 17:57 UTC

What happened:

Customers conducting live testing with Android Emulators, iOS Simulators and some desktop browsers in our EU-Central-1 data center experienced the session video continuously reloading when trying to interact with it, making live testing unusable.

Why it happened:

A collector component was deployed to the production environment. This component's main responsibility was to maintain the state of internal resources of the Sauce Labs platform. Due to a bug in one of the algorithms the state information of shared resources was removed. As a result a live testing component wasn’t able to operate properly.

How we fixed it:

The issue was resolved by the package rollback of the collector component.

What we are doing to prevent it from happening again:

We are Improving observability & monitoring of shared components and implementing better alerting based on discovered log anomalies within a specific time window of a configuration change event.

Posted Apr 01, 2021 - 09:17 UTC

Resolved

After taking remedial action, live testing is back to normal. All services are fully operational.

Posted Mar 23, 2021 - 17:57 UTC

Monitoring

We've taken remedial action and have seen live testing is back to normal functionality. We are monitoring

Posted Mar 23, 2021 - 17:48 UTC

Update

Live testing in EU-Central-1 data center is still experiencing issues. We continue to investigate

Posted Mar 23, 2021 - 17:34 UTC

Investigating

In Live Testing in our EU-Central-1 data center, the live video is intermittently reloading, causing the test session to be unusable. We are investigating.

Posted Mar 23, 2021 - 16:19 UTC

This incident affected: Live Browser Testing (EU-Central) and Live Virtual Mobile Device Testing (EU-Central).