Enlist fails at "hostname mass-enlist IP-Config: no response for n seconds giving up"

cityfan · 29 March 2024 23:28

Message repeats on client, but with hostname repeated i.e. “hostname mass-enlist hostname mass-enlist … IP-Config: no response after n seconds …”
Logs show status 200 for client pulling the boot-initrd and boot-kernel files but no request for the squashfs. Client eventually drops to a busybox shell that says “free raminitfs and switch to another rootfs”

snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.3.6-13230-g.3e02a33c3 33860 3.3/stable canonical✓ -

rackd.log
2024-03-29 18:45:32 provisioningserver.rackdservices.http: [info] pxelinux.cfg/default requested by 192.168.72.141
2024-03-29 18:45:32 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel requested by 192.168.72.141
2024-03-29 18:45:33 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd requested by 192.168.72.141

http/access.log
192.168.72.141 - - [29/Mar/2024:18:45:32 -0400] “GET /pxelinux.cfg/C HTTP/1.0” 404 0 “-” “Syslinux/6.04”
192.168.72.141 - - [29/Mar/2024:18:45:32 -0400] “GET /pxelinux.cfg/default HTTP/1.0” 200 1011 “-” “Syslinux/6.04”
192.168.72.141 - - [29/Mar/2024:18:45:33 -0400] “GET /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel HTTP/1.0” 200 11623336 “-” “Syslinux/6.04”
192.168.72.141 - - [29/Mar/2024:18:45:36 -0400] “GET /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd HTTP/1.0” 200 115824304 “-” “Syslinux/6.04”
::1 - - [29/Mar/2024:18:45:54 -0400] “GET /MAAS/rpc/ HTTP/1.1” 200 312 “-” “provisioningserver.rpc.clusterservice.ClusterClientService”
::1 - - [29/Mar/2024:18:46:24 -0400] “GET /MAAS/rpc/ HTTP/1.1” 200 312 “-” “provisioningserver.rpc.clusterservice.ClusterClientService”
::1 - - [29/Mar/2024:18:46:54 -0400] “GET /MAAS/rpc/ HTTP/1.1” 200 312 “-” “provisioningserver.rpc.clusterservice.ClusterClientService”

ls -lah /var/snap/maas/common/maas/boot-resources/snapshot-20240321-023851/ubuntu/amd64/ga-22.04/jammy/stable/
total 534M
drwxr-xr-x 2 root root 4.0K Mar 20 22:39 .
drwxr-xr-x 3 root root 4.0K Mar 20 22:39 …
-rw-r–r-- 3 root root 111M Mar 1 19:28 boot-initrd
-rw-r–r-- 3 root root 12M Mar 1 19:28 boot-kernel
-rw-r–r-- 8 root root 412M Mar 1 19:28 squashfs

cityfan · 31 March 2024 20:38

Backed up the postgresql 14 database with pg_dumpall and restored to a new Ubuntu 22.04 VM, then did a fresh snap install of maas channel=3.4. There were no errors reported installing maas with it pointing to the restored database, apart from invalid owner which I corrected with the ALTER DATABASE command.
The new instance shows all existing machines in the GUI, the same as the old version.

Trying to enlist machines with the new VM gives the same results as the old VM, logs show client machines download the boot-initrd and boot-kernel but there is no request for the squashfs file.

These are the details and log files from the new instance

snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.4.1-14343-g.a552d2522 34087 3.4/stable canonical✓ -

rackd.log
2024-03-31 20:12:34 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] ldlinux.c32 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/44454c4c-5200-1053-804c-c3c04f465031 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/01-84-2b-2b-70-d7-a9 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84898 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8489 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A848 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0 requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C requested by 192.168.72.152
2024-03-31 20:12:34 provisioningserver.rackdservices.http: [info] pxelinux.cfg/default requested by 192.168.72.152
2024-03-31 20:12:35 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel requested by 192.168.72.152
2024-03-31 20:12:35 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd requested by 192.168.72.152
2024-03-31 20:20:27 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: ens3, ens4.

ls -lah /var/snap/maas/common/maas/boot-resources/current/ubuntu/amd64/ga-22.04/jammy/stable
total 534M
drwxr-xr-x 2 root root 4.0K Mar 31 19:13 .
drwxr-xr-x 3 root root 4.0K Mar 31 19:13 …
-rw-r–r-- 3 root root 111M Mar 31 19:13 boot-initrd
-rw-r–r-- 3 root root 12M Mar 31 19:13 boot-kernel
-rw-r–r-- 8 root root 412M Mar 31 19:13 squashfs

troyanov · 1 April 2024 06:44

Hi @cityfan

Sounds like an issue with Ubuntu trying to boot. Can you share a bit more details about your VM?
Also does it work if you install Ubuntu on your VM without MAAS?

cityfan · 1 April 2024 14:03

Thanks very much for the reply troyanov.
The VM running MaaS as region+rack is stock Ubuntu 22.04 with 2 virtual NIC’s, both untagged. It is running on an Ubuntu 22.04 host physically connected to 2 switches, one for each NIC. The network segments are 192.168.72.0/24 and 172.22.254.0/24. The MaaS VM is at 192.168.72.3 and 172.22.254.3, and the bridges on the MaaS VM host are at 192.168.72.254 and 172.22.254.254.
MaaS is configured for DHCP on both networks.
Existing machines were PXE booted, provisioned and deployed via the 192.168.73.3 address and are visible in the MaaS GUI (both the 3.3 and new 3.4 deployments). PXE boot on the existing machines had VLAN disabled.
Traffic leaving the 192.168.72.0/24 switch to the rest of the network is tagged with VLAN 1687 (but as I said, not tagged at source).

We are trying to add 2 new machines in a separate physical segment. These 2 machines are on untagged ports on a separate switch and are configured at the source machines to PXE boot on VLAN 1687 and have IPMI over LAN also on VLAN 1687. One of these machines is at 192.168.72.152 in the logs above. Both machines display the same results, loading boot-kernel and boot-initrd but not squashfs.

I hope this makes sense, I can provide any other details that might help.
Thanks again for the reply.

cityfan · 1 April 2024 14:47

The 2 new machines are physical Dell servers. They were previously provisioned in this environment with Ubuntu 22.04 by MaaS and they can be booted without MaaS. They were deleted from MaaS and physically moved to a new location.

cityfan · 1 April 2024 15:52

The listing below shows the rackd.log for the 2 identical Dell R410’s that are PXE booting from our MaaS VM. The machine with address 192.168.72.153 succeeds and shows as “New” status in MaaS, the machine at address 192.168.72.152 fails to load squashfs.

Both machines (I’ll call them “WORKS” and “FAILS”) are connected to the same ethernet switch, the differences between them are the BIOS definitions and the ethernet switch port configurations.

WORKS has VLAN disabled in both the BMC and PXE boot configurations. The switch port it is attached to is expecting untagged traffic, and tags all traffic with VLAN 1687 to route it to MaaS.

FAILS has VLAN 1687 defined in the BMC and PXE boot configurations. The switch port it is attached to is expecting tagged traffic which should be routed to MaaS.

We want to get the FAILS configuration to work, so we can change the ethernet configuration to a bonded-pair and send all traffic over that pair separated by VLAN’s

rackd.log
2024-04-01 15:26:24 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] ldlinux.c32 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/44454c4c-5200-1053-804c-c3c04f465031 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/01-84-2b-2b-70-d7-a9 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84898 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8489 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A848 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0 requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] pxelinux.cfg/default requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel requested by 192.168.72.152
2024-04-01 15:26:24 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd requested by 192.168.72.152
2024-04-01 15:26:26 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=44598, flowInfo=0, scopeID=0) PEER:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=5253, flowInfo=0, scopeID=0))
2024-04-01 15:26:26 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=44602, flowInfo=0, scopeID=0) PEER:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=5253, flowInfo=0, scopeID=0))
2024-04-01 15:26:26 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=44610, flowInfo=0, scopeID=0) PEER:IPv6Address(type=‘TCP’, host=‘::ffff:172.22.254.3’, port=5253, flowInfo=0, scopeID=0))
2024-04-01 15:26:26 provisioningserver.rpc.clusterservice: [info] Event-loop ‘pdcmaasvm:pid=1300’ authenticated.
2024-04-01 15:26:27 provisioningserver.rpc.clusterservice: [info] Event-loop ‘pdcmaasvm:pid=1300’ authenticated.
2024-04-01 15:26:27 provisioningserver.rpc.clusterservice: [info] Event-loop ‘pdcmaasvm:pid=1300’ authenticated.
2024-04-01 15:26:27 provisioningserver.rpc.clusterservice: [info] Rack controller ‘fm67sn’ registered (via pdcmaasvm:pid=1300) with MAAS version 3.4.1-14343-g.a552d2522.
2024-04-01 15:26:27 provisioningserver.rpc.clusterservice: [info] Rack controller ‘fm67sn’ registered (via pdcmaasvm:pid=1300) with MAAS version 3.4.1-14343-g.a552d2522.
2024-04-01 15:26:27 provisioningserver.rpc.clusterservice: [info] Rack controller ‘fm67sn’ registered (via pdcmaasvm:pid=1300) with MAAS version 3.4.1-14343-g.a552d2522.
2024-04-01 15:26:51 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] ldlinux.c32 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/44454c4c-4200-1059-8044-c4c04f565631 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/01-90-b1-1c-09-1e-3d requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84899 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8489 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A848 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A84 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A8 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0A requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C0 requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/C requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] pxelinux.cfg/default requested by 192.168.72.153
2024-04-01 15:26:51 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel requested by 192.168.72.153
2024-04-01 15:26:52 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd requested by 192.168.72.153
2024-04-01 15:27:40 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/squashfs requested by 192.168.72.153

lavong · 12 April 2024 05:47

I have similar issue. All related to VLAN in the configuration.
Maas VM: 3.4.1-14343-g, rev 34087
Maas VM is tagged VLAN 123 on vnic. vSphere portgroup is trunk port which pass the Maas VM tagged traffic to VLAN 123.
Physical servers nic is VLAN 123 tagged on IPMI over LAN and PXE boot.
Maas able to commission host to power up, load ubuntu 22.04 GA image but failed IP-Config.

My guess, something to do with pxelinux.cfg must be configured with tagged VLAN so after PXE boot Ubuntu, it should abel to get IP on that network. I just don’t know where or if possible to do that.

any help appreciated.