Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2022-May-05 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Tuesday May 5th 2022, 2:00 AM - 7:39 AM UTC

What happened:

Real Devices in the EU-Central-1 datacenter were unable to use the internet and were displayed as “in use” to customers.

Why it happened:

The DHCP servers used by the devices crashed due to memory limitations being exceeded, which prevented the devices from connecting to the internet.

How we fixed it:

We restarted the DHCP daemon with blank lease files, allowing them to resynchronize quickly and begin answering requests for IP addresses.

What we are doing to prevent it from happening again:

We will optimize the way devices request leases from the DHCP servers, in order to minimize usage and we will add extra monitoring to our systems, while also making sure that we are running on the latest operating systems.

Posted May 11, 2022 - 09:12 UTC

Resolved

The Real Device issue has been resolved and all EU-Central-1 devices are available. Real Device tests are starting as normal with network connectivity on the devices restored.
Posted May 05, 2022 - 07:42 UTC

Investigating

All Real Devices in the eu-central-1 datacenter appear as IN USE. Tests will fail to start. We are investigating.
Posted May 05, 2022 - 07:00 UTC
This incident affected: Automated Real Device Testing (EU-Central), Live Real Device Testing (EU-Central), and Native Framework Mobile App Testing (EU-Central).