In the end it turned out that MAAS was set to keep these leases for a few weeks. So I had to manually remove the entries from dhcpd.services then restart maas-dhcpd.service:
sudo cp /var/lib/maas/dhcp/dhcpd.leases /var/lib/maas/dhcp/dhcpd.leases.save
sudo nano /var/lib/maas/dhcp/dhcpd.leases
sudo systemctl restart maas-dhcpd.service