Tuesday May 5th 2022, 2:00 AM - 7:39 AM UTC
Real Devices in the EU-Central-1 datacenter were unable to use the internet and were displayed as “in use” to customers.
The DHCP servers used by the devices crashed due to memory limitations being exceeded, which prevented the devices from connecting to the internet.
We restarted the DHCP daemon with blank lease files, allowing them to resynchronize quickly and begin answering requests for IP addresses.
We will optimize the way devices request leases from the DHCP servers, in order to minimize usage and we will add extra monitoring to our systems, while also making sure that we are running on the latest operating systems.