We’ve been using MAAS (v3.3.6) to manage machines in KVM using Virsh. We recently noticed an issue where new VMs get stuck in “Commissioning Powering node on”.
After digging a bit deeper it seems like the VLAN they are being assigned too has run out of addresses:
mcollins1@maas-rackd-01:~$ sudo journalctl -fu maas-dhcpd.service
...
Mar 21 11:43:57 maas-rackd-01 dhcpd[972535]: DHCPDISCOVER from 3a:71:38:3d:42:7d via extra-t-str
Mar 21 11:43:57 maas-rackd-01 dhcpd[972535]: DHCPOFFER on 10.241.0.11 to 3a:71:38:3d:42:7d via extra-t-str
Mar 21 11:44:40 maas-rackd-01 dhcpd[972535]: DHCPDISCOVER from 52:54:00:1e:62:08 via admin: network vlan-5060: no free leases
Mar 21 11:44:41 maas-rackd-01 dhcpd[972535]: DHCPDISCOVER from 52:54:00:1e:62:08 via admin: network vlan-5060: no free leases
Mar 21 11:44:43 maas-rackd-01 dhcpd[972535]: DHCPDISCOVER from 52:54:00:1e:62:08 via admin: network vlan-5060: no free leases
Mar 21 11:44:47 maas-rackd-01 dhcpd[972535]: DHCPDISCOVER from 52:54:00:1e:62:08 via admin: network vlan-5060: no free leases
The VLAN has available dynamic ranges on 146.118.58.2-10 and 146.118.58.31-50. When digging a little deeper it seems all of these addresses have been taken up by machines that were deleted a long time ago:
mcollins1@maas-rackd-01:~$ cat /var/lib/maas/dhcp/dhcpd.leases | grep -A 22 146.118.58 | grep 'set clht'
set clht = "ceph-t-rgw-01";
set clht = "storage-06-08110";
set clht = "pure-tetra";
set clht = "select-viper";
set clht = "immune-boxer";
set clht = "grown-hornet";
set clht = "causal-emu";
set clht = "choice-ewe";
set clht = "witty-stag";
set clht = "sunny-marmot";
set clht = "boss-amoeba";
set clht = "clear-shad";
set clht = "immune-hippo";
set clht = "living-skunk";
set clht = "simple-mutt";
set clht = "apt-dane";
set clht = "ceph-t-mon-01";
set clht = "ceph-t-mgr-01";
set clht = "ceph-t-kst-01";
set clht = "ceph-t-blc-01";
set clht = "ceph-t-rgw-01";
set clht = "ceph-t-mon-02";
set clht = "ceph-t-mem-01";
set clht = "ceph-t-kst-02";
set clht = "ceph-t-blc-02";
set clht = "ceph-t-rgw-02";
set clht = "ceph-t-mon-03";
set clht = "ceph-t-kst-03";
set clht = "ceph-t-blc-03";
Why isn’t MAAS cleaning up these leases properly? Has anyone experienced this before?
Also what is the proper way to handle this? Should we try deleting the leases manually from /var/lib/maas/dhcp/dhcpd.leases? Or should we perhaps run maas-dhcp-helper like so:
/usr/sbin/maas-dhcp-helper notify --action release --hostname "ceph-t-blc-03" --mac "52:54:0:92:4d:35" --ip-family ipv4 --ip "146.118.58.49" --socket /var/lib/maas/dhcpd.sock