Custom image deployment fails - 'curtin' curtin command install

Hi,

I am have a problem with deploying custom Ubuntu image. Image is generated by packer by other colleague. I uploaded image to MAAS by this command maas prof boot-resources create name='custom/ubuntu-es' title='Ubuntu ES' filetype='tgz' architecture='amd64/generic' content@=/home/admin/ubuntu-maas.tgz I tried also to include base_image=ubuntu/jammy but it had no effect.

Machine starts to deploy, but later it fails with messages in Event log:

 Mon, 11 Sep. 2023 09:35:41	Node changed status - From 'Deploying' to 'Failed deployment'
 Mon, 11 Sep. 2023 09:35:41	Marking node failed - Installation failed (refer to the installation log for more information).
 Mon, 11 Sep. 2023 09:35:41	Script - /tmp/curtin-logs.tar failed
 Mon, 11 Sep. 2023 09:35:41	Script - /tmp/install.log failed
 Mon, 11 Sep. 2023 09:35:41	Node installation failure - 'curtin' curtin command install
 Mon, 11 Sep. 2023 09:35:39	Node status event - 'curtin' configuring installed system
 Mon, 11 Sep. 2023 09:35:39	Node status event - 'curtin' running 'curtin curthooks'
 Mon, 11 Sep. 2023 09:35:39	Node status event - 'curtin' curtin command curthooks
 Mon, 11 Sep. 2023 09:35:39	Node status event - 'curtin' installing kernel

Installation output contains following part:

        Processing triggers for linux-image-5.15.0-83-generic (5.15.0-83.92) ...
        /etc/kernel/postinst.d/zz-update-grub:
        Sourcing file `/etc/default/grub'
        /usr/sbin/grub-mkconfig: 36: /etc/default/grub: Syntax error: Unterminated quoted string
        run-parts: /etc/kernel/postinst.d/zz-update-grub exited with return code 2
        dpkg: error processing package linux-image-5.15.0-83-generic (--configure):
         installed linux-image-5.15.0-83-generic package post-installation script subprocess returned error exit status 1
        Errors were encountered while processing:
         linux-image-5.15.0-83-generic
        needrestart is being skipped since dpkg has failed
        E: Sub-process /usr/bin/dpkg returned an error code (1)
        Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
        TIMED subp(['udevadm', 'settle']): 0.025
        Running command ['mount', '--make-private', '/tmp/tmp149ikpb_/target/sys'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmp149ikpb_/target/sys'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmp149ikpb_/target/run'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmp149ikpb_/target/run'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmp149ikpb_/target/proc'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmp149ikpb_/target/proc'] with allowed return codes [0] (capture=False)
        Running command ['mount', '--make-private', '/tmp/tmp149ikpb_/target/dev'] with allowed return codes [0] (capture=False)
        Running command ['umount', '/tmp/tmp149ikpb_/target/dev'] with allowed return codes [0] (capture=False)
        finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: FAIL: installing kernel
        finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
        Traceback (most recent call last):
          File "/curtin/curtin/commands/curthooks.py", line 380, in install_kernel
            map_suffix = mapping[codename][version]
        KeyError: 'jammy'
        
        During handling of the above exception, another exception occurred:
        
        Traceback (most recent call last):
          File "/curtin/curtin/commands/main.py", line 202, in main
            ret = args.func(args)
          File "/curtin/curtin/commands/curthooks.py", line 1886, in curthooks
            builtin_curthooks(cfg, target, state)
          File "/curtin/curtin/commands/curthooks.py", line 1728, in builtin_curthooks
            install_kernel(cfg, target)
          File "/curtin/curtin/commands/curthooks.py", line 385, in install_kernel
            distro.install_packages([kernel_fallback], target=target)
          File "/curtin/curtin/distro.py", line 424, in install_packages
            return install_cmd('install', args=pkglist, opts=opts, target=target,
          File "/curtin/curtin/distro.py", line 304, in run_apt_command
            cmd_rv = inchroot.subp(cmd, env=env)
          File "/curtin/curtin/util.py", line 787, in subp
            return subp(*args, **kwargs)
          File "/curtin/curtin/util.py", line 275, in subp
            return _subp(*args, **kwargs)
          File "/curtin/curtin/util.py", line 139, in _subp
            raise ProcessExecutionError(stdout=out, stderr=err,
        curtin.util.ProcessExecutionError: Unexpected error while running command.
        Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp149ikpb_/target', 'eatmydata', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', 'linux-generic']
        Exit code: 100
        Reason: -
        Stdout: ''
        Stderr: ''
        Unexpected error while running command.
        Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp149ikpb_/target', 'eatmydata', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', 'linux-generic']
        Exit code: 100
        Reason: -
        Stdout: ''
        Stderr: ''
        
Stderr: ''

I have no clue what could be wrong. Any help how to troubleshoot it is more than welcome :slight_smile: Thanks.

one issue was that image had no kernel. I have image with kernel an grub now, however I am still hitting some issue:

linux-image-5.15.0-82-generic is already the newest version (5.15.0-82.91).
The following packages were automatically installed and are no longer required:
  apport-symptoms python3-apport python3-problem-report python3-systemd
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
chattr: Operation not supported while setting flags on /tmp/tmpo20u87rz/target//swap.img
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=0ba5be93-d47e-4ef7-8f66-a5ac2c955ae2
Removing 'local diversion of /usr/sbin/update-initramfs to /usr/sbin/update-initramfs.curtin-disabled'
update-initramfs: Generating /boot/initrd.img-5.15.0-82-generic
Traceback (most recent call last):
  File "/tmp/tmpo20u87rz/target/curtin/curtin-hooks", line 59, in <module>
    main()
  File "/tmp/tmpo20u87rz/target/curtin/curtin-hooks", line 54, in main
    builtin_curthooks(config, target, state)
  File "/curtin/curtin/commands/curthooks.py", line 1851, in builtin_curthooks
    setup_grub(cfg, target, osfamily=osfamily,
  File "/curtin/curtin/commands/curthooks.py", line 804, in setup_grub
    install_grub(instdevs, target, uefi=uefi_bootable, grubcfg=grubcfg)
  File "/curtin/curtin/commands/install_grub.py", line 401, in install_grub
    in_chroot.subp(cmd, env=env, capture=True)
  File "/curtin/curtin/util.py", line 787, in subp
    return subp(*args, **kwargs)
  File "/curtin/curtin/util.py", line 275, in subp
    return _subp(*args, **kwargs)
  File "/curtin/curtin/util.py", line 139, in _subp
    raise ProcessExecutionError(stdout=out, stderr=err,
curtin.util.ProcessExecutionError: Unexpected error while running command.
Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpo20u87rz/target', 'dpkg-reconfigure', 'grub-pc']
Exit code: 2
Reason: -
Stdout: ''
Stderr: /var/lib/dpkg/info/grub-pc.config: 36: /etc/default/grub: Syntax error: Unterminated quoted string
        
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 202, in main
    ret = args.func(args)
  File "/curtin/curtin/commands/curthooks.py", line 1883, in curthooks
    if util.run_hook_if_exists(target, 'curtin-hooks'):
  File "/curtin/curtin/util.py", line 979, in run_hook_if_exists
    subp([target_hook])
  File "/curtin/curtin/util.py", line 275, in subp
    return _subp(*args, **kwargs)
  File "/curtin/curtin/util.py", line 139, in _subp
    raise ProcessExecutionError(stdout=out, stderr=err,
curtin.util.ProcessExecutionError: Unexpected error while running command.
Command: ['/tmp/tmpo20u87rz/target/curtin/curtin-hooks']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Unexpected error while running command.
Command: ['/tmp/tmpo20u87rz/target/curtin/curtin-hooks']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: start: cmd-install/stage-curthooks/builtin/cmd-curthooks: curtin command curthooks
        running /tmp/tmpo20u87rz/target/curtin/curtin-hooks
        Running command ['/tmp/tmpo20u87rz/target/curtin/curtin-hooks'] with allowed return codes [0] (capture=False)
        Adding 'local diversion of /usr/sbin/update-initramfs to /usr/sbin/update-initramfs.curtin-disabled'
        Running in chroot, ignoring command 'start'
        Hit:1 https://packages.wazuh.com/4.x/apt stable InRelease
        Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
        Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
        Get:4 http://archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
        Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
        Get:6 http://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [822 kB]
        Get:7 http://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B]
        Get:8 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [781 kB]
        Get:9 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB]
        Get:10 http://archive.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB]
        Get:11 http://archive.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B]
        Get:12 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [761 kB]
        Get:13 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB]
        Fetched 2767 kB in 1s (2486 kB/s)
        Reading package lists...
        Reading package lists...
        Building dependency tree...
        Reading state information...
        linux-image-5.15.0-82-generic is already the newest version (5.15.0-82.91).
        The following packages were automatically installed and are no longer required:
          apport-symptoms python3-apport python3-problem-report python3-systemd
        Use 'apt autoremove' to remove them.
        0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
        chattr: Operation not supported while setting flags on /tmp/tmpo20u87rz/target//swap.img
        Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
        no label, UUID=0ba5be93-d47e-4ef7-8f66-a5ac2c955ae2
        Removing 'local diversion of /usr/sbin/update-initramfs to /usr/sbin/update-initramfs.curtin-disabled'
        update-initramfs: Generating /boot/initrd.img-5.15.0-82-generic
        Traceback (most recent call last):
          File "/tmp/tmpo20u87rz/target/curtin/curtin-hooks", line 59, in <module>
            main()
          File "/tmp/tmpo20u87rz/target/curtin/curtin-hooks", line 54, in main
            builtin_curthooks(config, target, state)
          File "/curtin/curtin/commands/curthooks.py", line 1851, in builtin_curthooks
            setup_grub(cfg, target, osfamily=osfamily,
          File "/curtin/curtin/commands/curthooks.py", line 804, in setup_grub
            install_grub(instdevs, target, uefi=uefi_bootable, grubcfg=grubcfg)
          File "/curtin/curtin/commands/install_grub.py", line 401, in install_grub
            in_chroot.subp(cmd, env=env, capture=True)
          File "/curtin/curtin/util.py", line 787, in subp
            return subp(*args, **kwargs)
          File "/curtin/curtin/util.py", line 275, in subp
            return _subp(*args, **kwargs)
          File "/curtin/curtin/util.py", line 139, in _subp
            raise ProcessExecutionError(stdout=out, stderr=err,
        curtin.util.ProcessExecutionError: Unexpected error while running command.
        Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpo20u87rz/target', 'dpkg-reconfigure', 'grub-pc']
        Exit code: 2
        Reason: -
        Stdout: ''
        Stderr: /var/lib/dpkg/info/grub-pc.config: 36: /etc/default/grub: Syntax error: Unterminated quoted string
                
        finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
        Traceback (most recent call last):
          File "/curtin/curtin/commands/main.py", line 202, in main
            ret = args.func(args)
          File "/curtin/curtin/commands/curthooks.py", line 1883, in curthooks
            if util.run_hook_if_exists(target, 'curtin-hooks'):
          File "/curtin/curtin/util.py", line 979, in run_hook_if_exists
            subp([target_hook])
          File "/curtin/curtin/util.py", line 275, in subp
            return _subp(*args, **kwargs)
          File "/curtin/curtin/util.py", line 139, in _subp
            raise ProcessExecutionError(stdout=out, stderr=err,
        curtin.util.ProcessExecutionError: Unexpected error while running command.
        Command: ['/tmp/tmpo20u87rz/target/curtin/curtin-hooks']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: ''
        Unexpected error while running command.
        Command: ['/tmp/tmpo20u87rz/target/curtin/curtin-hooks']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: ''
        
Stderr: ''

Hmm, sticky wicket. Here are some suggestions on things I might try if I were in this pickle:

  • It looks like there may be an issue with the grub configuration in the image causing the deploy to fail. I would check the /etc/default/grub file and make sure it is valid syntax.
  • Also verify the image has the required kernel and bootloader installed. The errors mention it was missing the kernel initially.
  • When importing the image to MAAS, specify the architecture, release, and any other metadata like you would for Ubuntu images so curtin handles it correctly.
  • Inspect the curtin logs and /var/log/cloud-init.log on failed deployments for any other errors that could point to issues.
  • Try deploying the image on hardware, not just in MAAS, to rule out any virtualization-specific issues.
  • Look for a curtin preseed template file in the image that could be overriding the default config and causing problems.
  • Consider starting from a vanilla Ubuntu image and customizing that with Packer rather than building from scratch, so curtin/grub work out of the box.
  • Check the curtin source for examples of storage config files to properly define partitions.

Dunno if any of that helps, but maybe something in there is worth a shot.

Hi! The problem was with a typo in the grub config. Deployment proceeded after fixing it just fine.
Thanks for hints!