Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2022-May-05 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday May 5th 2022, 2:00 AM - 7:39 AM UTC
What happened:
Real Devices in the EU-Central-1 datacenter were unable to use the internet and were displayed as “in use” to customers.
Why it happened:
The DHCP servers used by the devices crashed due to memory limitations being exceeded, which prevented the devices from connecting to the internet.
How we fixed it:
We restarted the DHCP daemon with blank lease files, allowing them to resynchronize quickly and begin answering requests for IP addresses.
What we are doing to prevent it from happening again:
We will optimize the way devices request leases from the DHCP servers, in order to minimize usage and we will add extra monitoring to our systems, while also making sure that we are running on the latest operating systems.
Resolved
The Real Device issue has been resolved and all EU-Central-1 devices are available. Real Device tests are starting as normal with network connectivity on the devices restored.
Investigating
All Real Devices in the eu-central-1 datacenter appear as IN USE. Tests will fail to start. We are investigating.
This incident affected: Live Real Device Testing (EU-Central), Automated Real Device Testing (EU-Central), and Native Mobile App Testing (EU-Central).