Unable to locate configuration file

Users
,
1

Continuing the discussion from Unable to locate configuration file:

I thought this problem was resolved but we are experiencing this problem again.

logs from rack controller ( tail -f /var/snap/maas/common/log/rackd.log) seems to indicate the PXE server is fetching the file ( not sure why it’s there are calls for other files extensions)
2023-01-27 14:50:23 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.
2023-01-27 14:51:17 provisioningserver.rackdservices.tftp: [info] /pxelinux.0 requested by 10.166.105.144
2023-01-27 14:51:17 provisioningserver.rackdservices.tftp: [info] /pxelinux.0 requested by 10.166.105.144
2023-01-27 14:51:23 provisioningserver.rackdservices.tftp: [info] /ldlinux.c32 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/44454c4c-5a00-1032-8036-b2c04f515233 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/01-84-16-0c-e2-6f-a1 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA66990 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA6699 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA669 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA66 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA6 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0AA requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0A requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/0 requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/default requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/default requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.tftp: [info] /pxelinux.cfg/default requested by 10.166.105.144
2023-01-27 14:51:26 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-kernel requested by 10.166.105.144
2023-01-27 14:51:27 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd requested by 10.166.105.144
2023-01-27 14:52:15 provisioningserver.rackdservices.http: [info] /images/ubuntu/amd64/ga-22.04/jammy/stable/squashfs requested by 10.166.105.144

2

I have a packet capture file who do I upload? I tried uploading a *.pcap but that extension does not seem to be allowed.

including text dump

15:33:40.636164 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto UDP (17), length 576)

0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 84:16:0c:e2:6f:a1 (oui Unknown), length 548, xid 0xde26fa1, secs 4, Flags [Broadcast] (0x8000)

  Client-Ethernet-Address 84:16:0c:e2:6f:a1 (oui Unknown)

  Vendor-rfc1048 Extensions

    Magic Cookie 0x63825363

    DHCP-Message Option 53, length 1: Request

    Requested-IP Option 50, length 4: 10.166.105.144

    Parameter-Request Option 55, length 24:

      Subnet-Mask, Time-Zone, Default-Gateway, IEN-Name-Server

      Domain-Name-Server, RL, Hostname, BS

      Domain-Name, SS, RP, EP

      Vendor-Option, Server-ID, Vendor-Class, BF

      Option 128, Option 129, Option 130, Option 131

      Option 132, Option 133, Option 134, Option 135

    MSZ Option 57, length 2: 1260

    Server-ID Option 54, length 4: la-dhcp-1.somecompany.com

    GUID Option 97, length 17: 0.68.69.76.76.90.0.16.50.128.54.178.192.79.81.82.51

    ARCH Option 93, length 2: 0

    NDI Option 94, length 3: 1.2.1

    Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"

    END Option 255, length 0

    PAD Option 0, length 0, occurs 200

15:33:40.637254 IP (tos 0x0, ttl 128, id 13675, offset 0, flags [none], proto UDP (17), length 342)

la-dhcp-1.somecompany.com.bootps > 255.255.255.255.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 314, xid 0xde26fa1, Flags [none] (0x0000)

  Your-IP 10.166.105.144

  Server-IP la-maas-rack-01.somecompany.com

  Client-Ethernet-Address 84:16:0c:e2:6f:a1 (oui Unknown)

  file "/pxelinux.0"

  Vendor-rfc1048 Extensions

    Magic Cookie 0x63825363

    DHCP-Message Option 53, length 1: ACK

    RN Option 58, length 4: 345600

    RB Option 59, length 4: 604800

    Lease-Time Option 51, length 4: 691200

    Server-ID Option 54, length 4: la-dhcp-1.somecompany.com

    Subnet-Mask Option 1, length 4: 255.255.240.0

    Domain-Name-Server Option 6, length 8: 10.166.112.111,10.166.112.112

    Domain-Name Option 15, length 14: "somecompany.com^@"

    BF Option 67, length 12: "/pxelinux.0^@"

    END Option 255, length 0

15:33:40.638942 IP (tos 0x0, ttl 64, id 2, offset 0, flags [none], proto UDP (17), length 56)

10.166.105.144.2070 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  28 RRQ "/pxelinux.0" octet tsize 0

15:33:40.642458 IP (tos 0x0, ttl 64, id 4, offset 0, flags [none], proto UDP (17), length 61)

10.166.105.144.2071 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  33 RRQ "/pxelinux.0" octet blksize 1456

15:33:46.897506 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 70)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  42 RRQ "/ldlinux.c32" octet tsize 0 blksize 1408

15:33:47.038554 IP (tos 0x0, ttl 255, id 89, offset 0, flags [none], proto UDP (17), length 108)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  80 RRQ "/pxelinux.cfg/44454c4c-5a00-1032-8036-b2c04f515233" octet tsize 0 blksize 1408

15:33:47.109245 IP (tos 0x0, ttl 64, id 21678, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.109429 IP (tos 0x0, ttl 255, id 90, offset 0, flags [none], proto UDP (17), length 92)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  64 RRQ "/pxelinux.cfg/01-84-16-0c-e2-6f-a1" octet tsize 0 blksize 1408

15:33:47.171905 IP (tos 0x0, ttl 255, id 91, offset 0, flags [none], proto UDP (17), length 92)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  64 RRQ "/pxelinux.cfg/01-84-16-0c-e2-6f-a1" octet tsize 0 blksize 1408

15:33:47.180752 IP (tos 0x0, ttl 64, id 21688, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.180907 IP (tos 0x0, ttl 255, id 92, offset 0, flags [none], proto UDP (17), length 80)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  52 RRQ "/pxelinux.cfg/0AA66990" octet tsize 0 blksize 1408

15:33:47.180924 IP (tos 0x0, ttl 64, id 21689, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.181039 IP (tos 0x0, ttl 255, id 93, offset 0, flags [none], proto UDP (17), length 79)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  51 RRQ "/pxelinux.cfg/0AA6699" octet tsize 0 blksize 1408

15:33:47.181977 IP (tos 0x0, ttl 64, id 21690, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.182120 IP (tos 0x0, ttl 255, id 94, offset 0, flags [none], proto UDP (17), length 78)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  50 RRQ "/pxelinux.cfg/0AA669" octet tsize 0 blksize 1408

15:33:47.182720 IP (tos 0x0, ttl 64, id 21691, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.182816 IP (tos 0x0, ttl 255, id 95, offset 0, flags [none], proto UDP (17), length 77)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  49 RRQ "/pxelinux.cfg/0AA66" octet tsize 0 blksize 1408

15:33:47.184595 IP (tos 0x0, ttl 64, id 21692, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.184735 IP (tos 0x0, ttl 255, id 96, offset 0, flags [none], proto UDP (17), length 76)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  48 RRQ "/pxelinux.cfg/0AA6" octet tsize 0 blksize 1408

15:33:47.185324 IP (tos 0x0, ttl 64, id 21693, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.185419 IP (tos 0x0, ttl 255, id 97, offset 0, flags [none], proto UDP (17), length 75)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  47 RRQ "/pxelinux.cfg/0AA" octet tsize 0 blksize 1408

15:33:47.186835 IP (tos 0x0, ttl 64, id 21694, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.186934 IP (tos 0x0, ttl 255, id 98, offset 0, flags [none], proto UDP (17), length 74)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  46 RRQ "/pxelinux.cfg/0A" octet tsize 0 blksize 1408

15:33:47.187840 IP (tos 0x0, ttl 64, id 21695, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.187925 IP (tos 0x0, ttl 255, id 99, offset 0, flags [none], proto UDP (17), length 73)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  45 RRQ "/pxelinux.cfg/0" octet tsize 0 blksize 1408

15:33:47.189386 IP (tos 0x0, ttl 64, id 21696, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:33:47.189471 IP (tos 0x0, ttl 255, id 100, offset 0, flags [none], proto UDP (17), length 79)

10.166.105.144.49153 > la-maas-rack-01.somecompany.com.tftp: [udp sum ok]  51 RRQ "/pxelinux.cfg/default" octet tsize 0 blksize 1408

15:33:47.190129 IP (tos 0x0, ttl 64, id 21697, offset 0, flags [DF], proto UDP (17), length 47)

la-maas-rack-01.somecompany.com.tftp > 10.166.105.144.49153: [bad udp cksum 0xdf44 -> 0x1f62!]  19 ERROR ENOTFOUND "File not found"

15:40:03.145423 IP (tos 0x0, ttl 64, id 45422, offset 0, flags [DF], proto UDP (17), length 302)

la-maas-rack-01.somecompany.com.bootpc > 255.255.255.255.bootps: [bad udp cksum 0x6c0d -> 0x69f3!] BOOTP/DHCP, Request from 00:50:56:80:d2:c0 (oui Unknown), length 274, xid 0x76afa257, Flags [none] (0x0000)

  Client-Ethernet-Address 00:50:56:80:d2:c0 (oui Unknown)

  Vendor-rfc1048 Extensions

    Magic Cookie 0x63825363

    DHCP-Message Option 53, length 1: Discover

    Client-ID Option 61, length 23: "MAAS-00:50:56:80:d2:c0"

    Parameter-Request Option 55, length 3:

      Default-Gateway, Subnet-Mask, Domain-Name-Server

    END Option 255, length 0

cc @billwear

3

am i crazy, or are you somehow reading IP packets from the NIC before (and then after) the checksum is computed? that’s the only place i’ve seen this before. you might try checksum offloading to see if that helps?

4

Also, here’s the latest log entries.

2023-05-09 11:59:07 provisioningserver.kernel_opts: [debug] maas-enlist: kernel parameters — “nomodeset ro root=squash:http://10.166.115.22:5248/images/ubuntu/amd64/ga-22.04/jammy/stable/squashfs ip=::::maas-enlist:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{‘datasource_list’: [‘MAAS’]}end_cc cloud-config-url=http://10.166.115.22:5248/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed apparmor=0 log_host=10.166.115.22 log_port=5247”
2023-05-09 11:59:07 provisioningserver.kernel_opts: [debug] maas-enlist: kernel parameters — “nomodeset ro root=squash:http://10.166.115.22:5248/images/ubuntu/amd64/ga-22.04/jammy/stable/squashfs ip=::::maas-enlist:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:{‘datasource_list’: [‘MAAS’]}end_cc cloud-config-url=http://10.166.115.22:5248/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed apparmor=0 log_host=10.166.115.22 log_port=5247”
2023-05-09 11:59:12 provisioningserver.rpc.common: [debug] [RPC -> sent] AmpBox({b’uuid’: b’fafps6’, b’_command’: b’ListNodePowerParameters’})
2023-05-09 11:59:13 provisioningserver.rpc.common: [debug] [RPC <- recieved] AmpBox({b’_answer’: b’1f5a’, b’nodes’: b’’})
2023-05-09 11:59:13 tftp.bootstrap: [debug] Timed out during option negotiation process

5

Hey, Clarence!

Looks like you’re still fighting that issue with reading IP packets and hitting a timeout during option negotiation. I tried to come up with few more suggestions that might help you troubleshoot the problem:

  1. Checksum Offloading: Well, actually you’ve heard this one before. Did you try enabling checksum offloading on your network interface card (NIC)? Might help with packet processing.
  2. Network Configuration: Ensure that your network configuration is set up correctly, including the proper IP settings, subnet masks, and default gateway. Double-check if the IP address specified in the kernel parameters is accurate and reachable.
  3. DHCP and TFTP Configuration: Verify that the DHCP and TFTP services are properly configured and running on your rack controller. Check the relevant configuration files and logs to ensure that they are set up correctly and not encountering any errors. Some people say, “it’s always DNS”, but with MAAS, “it’s always DHCP” (and/or not configuring your external DHCP next-server with the IP of the TFTP server).
  4. Packet Capture: If you don’t get any joy here, packet capture and Wireshark might provide more insights. You probably already know how to analyze the captured packets to identify any anomalies or errors during the communication process.
  5. Community Support: Keep the topic hot. Keep asking. Consider reaching out to the community again (and again) or seeking assistance from other MAAS experts in the community. If you flip through the posts, you might find someone who had similar issues, and maybe you can ping them directly by handle. Hey, you never know, maybe they can provide more specific guidance tailored to your setup and configuration.

Please keep in mind a couple of things: (1) I’m just the technical author, so I’m sort of a power user, not a MAAS developer (though I can read the code); (2) these suggestions are general troubleshooting steps, not direct answers. Good luck finding this. Let me know how else I can help.