Error deploying RHEL 8 image built using packer from MAAS 3.2 snap install

bedfordc · 4 August 2022 15:57

I am trying to deploy a RHEL8 instance on a new Dell PowerEdge R6515, and it seems to be working until after the disks are set up and curtin attempts to chroot to the new environment to install packages.

It throws the following messages:
```
Finalizing /tmp/tmpxcp08afk/target
finish: cmd-install/stage-hook/builtin/cmd-hook: SUCCESS: curtin command hook
start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: curtin command in-target
Running command [‘mount’, ‘–bind’, ‘/dev’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/proc’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/run’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/sys’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/sys/firmware/efi/efivars’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘unshare’, ‘–help’] with allowed return codes [0] (capture=True)
Running command [‘unshare’, ‘–fork’, ‘–pid’, ‘–’, ‘chroot’, ‘/tmp/tmpxcp08afk/target’, ‘bash’, ‘-c’, ‘dpkg-query -s cloud-init || (echo “cloud-init not detected, MAAS will not be able to configure this machine properly” && exit 1)’] with allowed return codes [0] (capture=False)
bash: dpkg-query: command not found
cloud-init not detected, MAAS will not be able to configure this machine properly
Running command [‘udevadm’, ‘settle’] with allowed return codes [0] (capture=False)
TIMED subp([‘udevadm’, ‘settle’]): 0.005
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: FAIL: curtin command in-target
curtin: Installation failed with exception: Unexpected error while running command.
Command: [‘curtin’, ‘in-target’, ‘–’, ‘bash’, ‘-c’, ‘dpkg-query -s cloud-init || (echo “cloud-init not detected, MAAS will not be able to configure this machine properly” && exit 1)’]
Exit code: 1
Reason: -
Stdout: start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: curtin command in-target
Running command [‘mount’, ‘–bind’, ‘/dev’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/proc’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/run’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/sys’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–bind’, ‘/sys/firmware/efi/efivars’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘unshare’, ‘–help’] with allowed return codes [0] (capture=True)
Running command [‘unshare’, ‘–fork’, ‘–pid’, ‘–’, ‘chroot’, ‘/tmp/tmpxcp08afk/target’, ‘bash’, ‘-c’, ‘dpkg-query -s cloud-init || (echo “cloud-init not detected, MAAS will not be able to configure this machine properly” && exit 1)’] with allowed return codes [0] (capture=False)
bash: dpkg-query: command not found
cloud-init not detected, MAAS will not be able to configure this machine properly
Running command [‘udevadm’, ‘settle’] with allowed return codes [0] (capture=False)
TIMED subp([‘udevadm’, ‘settle’]): 0.005
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/sys/firmware/efi/efivars’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/sys’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/run’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/proc’] with allowed return codes [0] (capture=False)
Running command [‘mount’, ‘–make-private’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
Running command [‘umount’, ‘/tmp/tmpxcp08afk/target/dev’] with allowed return codes [0] (capture=False)
finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: FAIL: curtin command in-target

Stderr: ''
```

I verified that cloud-init is included in the rhel8.ks file in the packer image.

Any direction on how to get past this would be greatly appreciated.

Thanks

~~ Charles

bedfordc · 8 August 2022 16:00

After digging in a bit more, I tried to install rhel8 on a second host. This one had a slightly different experience, in that I was able to see on the console that it couldn’t boot to the image it was trying to boot into, probably because MAAS was pointing the boot process to the wrong location.

Thoughts?

Anyway - here is the console screen-shot, and the tail of the logs:

Finalizing /tmp/tmpplouiu3_/target
finish: cmd-install/stage-hook/builtin/cmd-hook: SUCCESS: curtin command hook
start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: curtin command in-target
Running command ['mount', '--bind', '/dev', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/proc', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/run', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
Running command ['unshare', '--help'] with allowed return codes [0] (capture=True)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpplouiu3_/target', 'bash', '-c', 'dpkg-query -s cloud-init || (echo "cloud-init not detected, MAAS will not be able to configure this machine properly" && exit 1)'] with allowed return codes [0] (capture=False)
bash: dpkg-query: command not found
cloud-init not detected, MAAS will not be able to configure this machine properly
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.009
Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: FAIL: curtin command in-target
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'in-target', '--', 'bash', '-c', 'dpkg-query -s cloud-init || (echo "cloud-init not detected, MAAS will not be able to configure this machine properly" && exit 1)']
Exit code: 1
Reason: -
Stdout: start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: curtin command in-target
        Running command ['mount', '--bind', '/dev', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--bind', '/proc', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--bind', '/run', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--bind', '/sys', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
        Running command ['unshare', '--help'] with allowed return codes [0] (capture=True)
        Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpplouiu3_/target', 'bash', '-c', 'dpkg-query -s cloud-init || (echo "cloud-init not detected, MAAS will not be able to configure this machine properly" && exit 1)'] with allowed return codes [0] (capture=False)
        bash: dpkg-query: command not found
        cloud-init not detected, MAAS will not be able to configure this machine properly
        Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
        TIMED subp(['udevadm', 'settle']): 0.009
        Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmpplouiu3_/target/sys'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmpplouiu3_/target/run'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmpplouiu3_/target/proc'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmpplouiu3_/target/dev'] with allowed return codes [0] (capture=False)
        finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: FAIL: curtin command in-target

Thanks in advance

~~ Charles

billwear · 15 August 2022 19:55

these both appear to be failing on the same error: curtin is checking that cloud-init is installed, and it isn’t, so curtin won’t finish. i don’t yet know why this is happening, but i’m trying to figure it out.

billwear · 15 August 2022 20:08

are you absolutely sure that (1) you packed the RHEL8 image correctly, and (2) you uploaded it to MAAS? this looks suspiciously like cloud-init is trying to prepare for an Ubuntu image, and your console image seems to indicate that your image didn’t get uploaded or isn’t recognized as valid:

Loading http://10.244.140.33:52348/images/cusdtom/amd64/ga-20.04/rhel8/no-such-image/...

can you share the command line you used to upload the RHEL 8 image to MAAS?

bedfordc · 16 August 2022 07:44

My command to load MAAS with rhel8 is as follows:

maas admin boot-resources create name='custom/rhel8' title="RedHat Enterprise Linux 8" architecture='amd64/generic' filetype=tgz content@=rhel8.tar.gz

That is similar to my conclusions as well. When it’s checking to see if cloud-init is installed it’s using ubuntu tools to find out - which don’t work on rhel - they should say rpm -q instead of dpkg-query. Which of course is going to ilicit an error.

Also the variables that should be set are not - as evidenced by the path to the image detailed by the root= part of the nomodeset line in the boot image.

I was following the instructions in the README.TXT file that came in the git clone from canonical’s rhel8 packer-maas.git for loading it back into MAAS. Note: it was built on a baremetal host which was different than the vmware based maas server - then scp’d over for getting loaded.

Thanks

~~ Charles

bedfordc · 24 August 2022 12:37

Any new thoughts @billwear?

dholt · 24 August 2022 20:35

I’m having a similar issue. It seems like it changed with MAAS 3.1 requiring the base_image parameter.

maas admin boot-resources create name='custom/dgx_a100-5.4' title='NVIDIA DGX A100 5.4' architecture='amd64/generic' filetype='tgz' base_image='ubuntu/focal' content@=dgxos5.tar.gz

$ ls /var/lib/maas/boot-resources/current/custom/amd64/generic/dgx_a100-5.4/uploaded/
root-tgz

bedfordc · 25 August 2022 07:30

Sounds like a bug to me.

Perhaps I should open one on the bugtracker?

Thanks

~~ Charles

bedfordc · 25 August 2022 07:53

bedfordc:

        Running command ['unshare', '--help'] with allowed return codes [0] (capture=True)
        Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpplouiu3_/target', 'bash', '-c', 'dpkg-query -s cloud-init || (echo "cloud-init not detected, MAAS will not be able to configure this machine properly" && exit 1)'] with allowed return cod

I’ve submitted the bug here: https://bugs.launchpad.net/maas/+bug/1987622

Thanks

bedfordc · 29 August 2022 08:59

Alberto was able to give me the fix - which was to add an additional parameter to the CLI command I used to load the image into MAAS. The required paramter was base_image=“rhel/rhel8” in this case.

It is probably something that should be included in the documentation…

Thanks everyone for your assistance.

~~ Charles

system · 31 August 2022 08:59

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.

billwear · 7 September 2022 16:57

sorry, been on the “august holiday” – i’ll try to take a look at this again soon.