Uefi_find_duplicate_entries issue caused by bug in Intel BIOS

Users
1

I have an aggravating issue because I have 100 old servers and Intel’s EFI BIOS has a bug.

The bug is Intel uses lower case letters when creating EFI boot entries when the spec says to use uppercase. For example Boot000a instead of Boot000A. (these are hexidecimal indexes).

MAAS throws an error in uefi_find_duplicate_entries “KeyError” because it is looking for 000A and it’s not there.

I’m running maas 3.2.7-12037-g.c688dd446

Has anyone else seen anything similar ?

Is there a way to create a custom version of /snap/maas/26274/usr/lib/python3/dist-packages/curtin/commands/curthooks.py ?? Is this even where the file that’s run on the host being deployed comes from or does that file come from an ubuntu package? I’d like to try to find a work around. In the meantime, I am stripping all machines down to less than 10 boot devices, but they sometimes seem to get recreated and I don’t know why, which makes commissioning painfully manual.

2

Hi @jeremy-mordkoff

I’ve checked curtin codebase and it seems that KeyError might indeed happen only when there is come inconsistency in boot entries names.

Can you please paste the output of efibootmgr -v here?

Regarding your question of how to create a custom version of curtin and which hook runs on the machine:

During deployment cloud-init will download a packed curtin as part of a user-data payload.
Since you have a snap installation, MAAS will on the fly pack curtin that comes as a MAAS dependency (python3-curtin), however in deb installation it will pick up curtin that is installed on your machine (where MAAS is configured). So yes, the file you’ve mentioned is the one being executed on the target maching.

I think that the easiest way to modify it is to unsquash snap package, do the changes and then do snap try:

For example:

sudo unsquashfs maas_XXXX.snap 
sudo snap try squashfs-root/

And don’t forget to check snap connections, otherwise weird things are possible.

Hope that will help you to solve you problem.

3

This only affects certain models that use a particular Intel BIOS version. I have already verified that we are running the latest BIOS but these machines are old and the BIOS has not been updated in years.

root@juju:~# juju ssh 1 -- efibootmgr -v 
** Warning ** : Boot001a is not UEFI Spec compliant (lowercase hex in name)
** Warning ** : please recreate these using efibootmgr to remove this warning.
BootCurrent: 0011
Timeout: 5 seconds
BootOrder: 0011,0000,0015,0010,0016,0005,0017,0018,0019,001A
Boot0000* ubuntu        HD(1,GPT,2d61ff62-b083-4152-8267-2eedc1324435,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)
Boot0005* Internal EFI Shell    VenMedia(5023b95c-db26-429b-a648-bd47664c8012)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
Boot0010* UEFI IPv4: Network Card 8701  PciRoot(0x40)/Pci(0x3,0x0)/Pci(0x0,0x1)/MAC(6cb3113b4035,1)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0011* UEFI IPv4: Network Card 8101  PciRoot(0x40)/Pci(0x1,0x0)/Pci(0x0,0x1)/MAC(6cb3113b40bd,1)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0015* UEFI IPv4: Network Card 8700  PciRoot(0x40)/Pci(0x3,0x0)/Pci(0x0,0x0)/MAC(6cb3113b4033,1)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0016* UEFI IPv4: Network Card 8100  PciRoot(0x40)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(6cb3113b40bb,1)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0017* UEFI IPv4: Network Card 0400  PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/MAC(001e6749d6aa,0)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0018* UEFI IPv4: Network Card 0401  PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x1)/MAC(001e6749d6ab,0)/IPv4(0.0.0.00.0.0.0,0,0)
Boot0019* UEFI IPv4: Network Card 0402  PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x2)/MAC(001e6749d6ac,0)/IPv4(0.0.0.00.0.0.0,0,0)
Boot001a* UEFI IPv4: Network Card 0403  PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x3)/MAC(001e6749d6ad,0)/IPv4(0.0.0.00.0.0.0,0,0)
Connection to 10.157.97.100 closed.
4

Hey, @jeremy-mordkoff,

This one’s sitting in a weird state, so I’m sticking my nose in, hopefully in a helpful way. This post seems to be discussing an issue with uppercase vs lowercase EFI boot entries causing problems with MAAS deployment. But the conversation ends kinda abruptly. Let me spitball some ideas for moving this forward:

  • First, thanks for reporting this issue and providing details on the efibootmgr output. I understand how inconsistencies with EFI boot entry names could cause problems (read “be a real PITA”).
  • Would you be able to provide more details on exactly where in the MAAS deployment process the issue is occurring? Are you seeing failures at commissioning, deployment, etc?
  • Additional logs from the MAAS server and client would help troubleshoot further. The curtin logs may also contain hints.
  • As a workaround, you could try recreating the EFI entries with consistent uppercase names using efibootmgr first. But I understand this may not be feasible with many machines.
  • This seems like a bug that the MAAS team should look into further. I would suggest opening a bug report on Launchpad with details to get their input.
  • Please let me know if you have any other info on where in the process the failures occur. More details will help troubleshoot.
  • For now, suggestions on where exactly this arises and extra logs would be great.

Let me know if you have any other details to share! I’m happy to help troubleshoot further or, at least, help get the issue properly logged.

5

I should be able to reproduce this one. It happens during commissioning (I think).

1 Like