Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2022-May-05 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday May 5th 2022, 2:00 AM - 7:39 AM UTC
What happened:
Real Devices in the EU-Central-1 datacenter were unable to use the internet and were displayed as “in use” to customers.
Why it happened:
The DHCP servers used by the devices crashed due to memory limitations being exceeded, which prevented the devices from connecting to the internet.
How we fixed it:
We restarted the DHCP daemon with blank lease files, allowing them to resynchronize quickly and begin answering requests for IP addresses.
What we are doing to prevent it from happening again:
We will optimize the way devices request leases from the DHCP servers, in order to minimize usage and we will add extra monitoring to our systems, while also making sure that we are running on the latest operating systems.
Resolved
The Real Device issue has been resolved and all EU-Central-1 devices are available. Real Device tests are starting as normal with network connectivity on the devices restored.
Investigating
All Real Devices in the eu-central-1 datacenter appear as IN USE. Tests will fail to start. We are investigating.
This incident affected: Automated Real Device Testing (EU-Central), Live Real Device Testing (EU-Central), and Native Framework Mobile App Testing (EU-Central).