Dates

Tuesday, March 23, 11:07 - 17:57 UTC

What happened:

Customers conducting live testing with Android Emulators, iOS Simulators and some desktop browsers in our EU-Central-1 data center experienced the session video continuously reloading when trying to interact with it, making live testing unusable.

Why it happened:

A collector component was deployed to the production environment. This component's main responsibility was to maintain the state of internal resources of the Sauce Labs platform. Due to a bug in one of the algorithms the state information of shared resources was removed. As a result a live testing component wasn’t able to operate properly. 

How we fixed it:

The issue was resolved by the package rollback of the collector component.

What we are doing to prevent it from happening again:

We are Improving observability & monitoring of shared components and implementing better alerting based on discovered log anomalies within a specific time window of a configuration change event.