Some DHCP leases don't update DNS

[home]

We have used MAAS for over 3 years to deploy and manage a hundred or so physical and virtual machines. We recently started changing from DHCP to Auto assign and as such reduced the DHCP pools. Machines which had IPs in those pools were assigned new IPs at next renewal time by the DHCP server, as expected. These machines are still operating fine, but DNS is not updated properly.

I see that /etc/bind/maas/zone.OUR-ZONE has the wrong entries. If I manually update them then restart maas-rackd and maas-regiond, they revert to the old wrong contents. This makes me suspect the database. I had a hunt around and found for example found the entries below.

cinder-t-02 had 146.118.53.240 until recently, and was then assigned 146.118.53.131 then not long later 146.118.53.144 where it is communicating fine. The DNS entry however still points to .240.

May I have some help troubleshooting this? There are at least a dozen more nodes affected.

Thanks,
Greg.

root@maas:~# ip=146.118.53.240
root@maas:~# grep -E "(Data for Name:|^COPY|$ip)" ~/pgdump.sql | grep -B2 $ip

-- Data for Name: maasserver_neighbour; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_neighbour (id, created, updated, ip, "time", vid, count, mac_address, interface_id) FROM stdin;
46565   2020-11-24 17:32:05.526191+00   2020-12-01 22:02:01.361616+00   146.118.53.240  1606860121      \N      17      52:54:00:7a:9b:ee       6370
--
-- Data for Name: maasserver_rdns; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_rdns (id, created, updated, ip, hostname, hostnames, observer_id) FROM stdin;
26289   2020-11-24 17:32:05.869085+00   2020-12-01 22:02:01.785686+00   146.118.53.240  cinder-t-02.OUR-ZONE        ["cinder-t-02.OUR-ZONE"]    1
--
-- Data for Name: maasserver_staticipaddress; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_staticipaddress (id, created, updated, ip, alloc_type, subnet_id, user_id, lease_time, temp_expires_on) FROM stdin;
58574   2020-12-01 05:31:55+00  2020-12-01 05:31:55.58178+00    146.118.53.240  6       1       \N      86400   \N


root@maas:~# ip=146.118.53.131
root@maas:~# grep -E "(Data for Name:|^COPY|$ip)" ~/pgdump.sql | grep -B2 $ip
-- Data for Name: maasserver_neighbour; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_neighbour (id, created, updated, ip, "time", vid, count, mac_address, interface_id) FROM stdin;
44856   2019-02-25 04:15:42.876927+00   2019-02-25 07:48:12.471091+00   146.118.53.131  1551080892      \N      3       52:54:00:d7:86:69       6406
47017   2020-12-02 05:31:54.807908+00   2020-12-02 07:55:57.1051+00     146.118.53.131  1606895757      \N      5       52:54:00:7a:9b:ee       6370
--
-- Data for Name: maasserver_rdns; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_rdns (id, created, updated, ip, hostname, hostnames, observer_id) FROM stdin;
26762   2020-12-02 05:31:55.298696+00   2020-12-02 07:55:57.285173+00   146.118.53.131  146-118-53-131.maas     ["146-118-53-131.maas"] 1


root@maas:~# ip=146.118.53.144
root@maas:~# grep -E "(Data for Name:|^COPY|$ip)" ~/pgdump.sql | grep -B2 $ip
-- Data for Name: maasserver_dnspublication; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_dnspublication (id, serial, created, source) FROM stdin;
878458  878460  2020-12-01 22:33:38.944155+00   ip 146.118.53.144 disconnected from testing-heat-t-03 on ens4
878459  878461  2020-12-01 22:33:38.944155+00   ip 146.118.53.144 disconnected from testing-heat-t-03 on br-fw-admin
--
-- Data for Name: maasserver_neighbour; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_neighbour (id, created, updated, ip, "time", vid, count, mac_address, interface_id) FROM stdin;
47039   2020-12-02 08:02:21.295569+00   2020-12-02 09:19:23.711761+00   146.118.53.144  1606900763      \N      10      52:54:00:7a:9b:ee       6370
--
-- Data for Name: maasserver_rdns; Type: TABLE DATA; Schema: public; Owner: maas
COPY public.maasserver_rdns (id, created, updated, ip, hostname, hostnames, observer_id) FROM stdin;
26784   2020-12-02 08:12:23.284919+00   2020-12-02 09:19:23.86751+00    146.118.53.144  146-118-53-144.maas     ["146-118-53-144.maas"] 1

[billwear]

@home, what version of MAAS are you using?

[gregoryo2017]

Now I’ve got my main Ubuntu One account working properly, I’m logged in there. @billwear we’re using 2.8, upgraded over the years from whatever was available in Jan 2017.
Thanks,
Greg.

[gregoryo2017]

And now, bizarrely, the DNS updates have started working. I wonder about this:

Dec  3 04:23:22 maas sh[12295]: 2020-12-03 04:23:22 maasserver.rpc.leases: [info] Lease update: commit for 146.118.53.144 on 52:54:0:7a:9b:ee at 2020-12-03 04:23:22 (lease time: 86400s) (hostname: cinder-t-02)
Dec  3 04:23:23 maas sh[12295]: 2020-12-03 04:23:23 maasserver.dhcp: [warn] No DNS servers found, DHCP defaulting to region IP.

And then this:

Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[1] = notify
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[2] = --action
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[3] = commit
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[4] = --mac
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[5] = 52:54:0:7a:9b:ee
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[6] = --ip-family
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[7] = ipv4
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[8] = --ip
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[9] = 146.118.53.144
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[10] = --lease-time
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[11] = 86400
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[12] = --hostname
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[13] = cinder-t-02
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[14] = --socket
Dec  3 23:54:20 maas dhcpd[14624]: execute_statement argv[15] = /var/lib/maas/dhcpd.sock
Dec  3 23:54:20 maas dhcpd[14624]: Wrote 0 class decls to leases file.
Dec  3 23:54:20 maas dhcpd[14624]: Wrote 0 deleted host decls to leases file.
Dec  3 23:54:20 maas dhcpd[14624]: Wrote 0 new dynamic host decls to leases file.
Dec  3 23:54:20 maas sh[14102]: 2020-12-03 23:54:20 maasserver.rpc.leases: [info] Lease update: commit for 146.118.53.144 on 52:54:0:7a:9b:ee at 2020-12-03 23:54:20 (lease time: 86400s) (hostname: cinder-t-02)
Dec  3 23:54:20 maas dhcpd[14624]: Wrote 1053 leases to leases file.
Dec  3 23:54:20 maas dhcpd[14624]: DHCPREQUEST for 146.118.53.144 from 52:54:00:7a:9b:ee (cinder-t-02) via ens3
Dec  3 23:54:20 maas dhcpd[14624]: DHCPACK on 146.118.53.144 to 52:54:00:7a:9b:ee (cinder-t-02) via ens3

And it seems to have started working, perhaps around this time but we haven’t been checking continuously.

[billwear]

@home, it happens: as soon as you tell someone else about it, the bug goes away. we’ll be here if you still have trouble.