Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2021-June-10 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday, June 10th, 06:13 - 08:58 UTC

What happened:

Android real devices had elevated error rates and some unavailability in our US-West-1 and EU-Central-1 data centers, causing some tests to fail.

Why it happened:

Memory spikes were observed in our Android device pools. We also found that two sets of a device pool pod were running, causing some internal endpoint failures. 

How we fixed it:

The remediation here was to reboot the hardware hosting the affected service, which resolved the issue with the multiple pods running. To address the memory spikes, we rolled back to a previously deployed RDC service version.

What we are doing to prevent it from happening again:

We are analyzing the code changes that could have caused this to learn how we can avoid this issue in the future.

Posted Jun 18, 2021 - 09:43 UTC

Resolved

All eu-central-1 and us-west-1 RDC Android tests should be operating normally.
Posted Jun 10, 2021 - 14:24 UTC

Update

We have taken remedial action and are monitoring the state of RDC Android testing in us-west-1 and eu-central-1.
Posted Jun 10, 2021 - 10:49 UTC

Monitoring

RDC Android devices are experiencing elevated error rates and unavailability in us-west-1 and eu-central-1. We are taking remedial action. RDC Android tests may fail during this time.
Posted Jun 10, 2021 - 09:04 UTC
This incident affected: Automated Real Device Testing (US-West, EU-Central).