Deploy fails during step 'installing kernel'

trsoumi88 · 24 June 2024 14:23

Hi,

I created a custom image, which had kernel ‘5.4.0.177.175’. An attempt to deploy (**MaaS version = 3.2.10 **) the image on a machine failed during step ‘installing kernel’ - with package dependency error as shown below. The error was thrown after the curom image was successfully copied to the disk.

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: installing kernel
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp4k_jgx77/target', 'lsb_release', '--all'] with allowed return codes [0] (capture=True)
Running command ['dpkg', '--print-architecture'] with allowed return codes [0] (capture=True)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp4k_jgx77/target', 'lsb_release', '--codename', '--short'] with allowed return codes [0] (capture=True)
Couldn't detect kernel package to install for 5.4.0-182-generic.
Running command ['mount', '--bind', '/dev', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/proc', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/run', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys/firmware/efi/efivars', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp4k_jgx77/target', 'apt-get', '--quiet', '--option=Acquire::Languages=none', '--option=Dir::Etc::sourcelist=/tmp/tmp6_3n2k4t/sources.list', '--option=Dir::Etc::sourceparts=/tmp/tmp6_3n2k4t/sources.list.d', 'update'] with allowed return codes [0] (capture=False)
Running in chroot, ignoring request: start
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease [1189 B]
Get:4 http://archive.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Get:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  Packages [16.3 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [970 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/main i386 Packages [718 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 c-n-f Metadata [29.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/universe i386 Packages [4642 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [8628 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 c-n-f Metadata [265 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/restricted i386 Packages [8112 B]
Get:14 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [22.0 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal/restricted amd64 c-n-f Metadata [392 B]
Get:16 http://archive.ubuntu.com/ubuntu focal/multiverse i386 Packages [74.7 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [144 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 c-n-f Metadata [9136 B]
Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3361 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [986 kB]
Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB]
Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1194 kB]
Get:23 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [788 kB]
Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB]
Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2996 kB]
Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B]
Get:28 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8440 B]
Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB]
Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B]
Get:31 http://archive.ubuntu.com/ubuntu focal-security/main i386 Packages [763 kB]
Get:32 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [2985 kB]
Get:33 http://archive.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB]
Get:34 http://archive.ubuntu.com/ubuntu focal-security/universe i386 Packages [661 kB]
Get:35 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [967 kB]
Get:36 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB]
Get:37 http://archive.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2863 kB]
Get:38 http://archive.ubuntu.com/ubuntu focal-security/restricted i386 Packages [36.4 kB]
Get:39 http://archive.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B]
Get:40 http://archive.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.0 kB]
Get:41 http://archive.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7200 B]
Get:42 http://archive.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B]
Get:43 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB]
Get:44 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB]
Get:45 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1420 B]
Get:46 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB]
Get:47 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB]
Get:48 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B]
Get:49 http://archive.ubuntu.com/ubuntu focal-backports/restricted amd64 c-n-f Metadata [116 B]
Get:50 http://archive.ubuntu.com/ubuntu focal-backports/multiverse amd64 c-n-f Metadata [116 B]
Fetched 34.1 MB in 7s (4567 kB/s)
Reading package lists...
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.012
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/dev', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/proc', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/run', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys/firmware/efi/efivars', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmp4k_jgx77/target', 'eatmydata', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', 'linux-generic'] with allowed return codes [0] (capture=False)
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-generic : Depends: linux-image-generic (= 5.4.0.186.184) but 5.4.0.177.175 is to be installed
                 Depends: linux-headers-generic (= 5.4.0.186.184) but 5.4.0.177.175 is to be installed
E: Unable to correct problems, you have held broken packages.
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.019
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/sys/firmware/efi/efivars'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmp4k_jgx77/target/dev'] with allowed return codes [0] (capture=False)
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: FAIL: installing kernel

I tried to troubleshoot further in Rescue mode. I could recreate the issue, by running the command (from installation logs) on the mounted partition:

root@localhost:~# TPM_DIR="/mnt"

root@localhost:~# unshare  --fork --pid -- chroot "${TMP_DIR}" lsb_release --all
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
root@localhost:~#

root@localhost:~# unshare  --fork --pid -- chroot "${TMP_DIR}" apt update -y
Running in chroot, ignoring request: start
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Hit:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3359 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [986 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1194 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [788 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [287 kB]
Fetched 6743 kB in 5s (1407 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
29 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@localhost:~# unshare  --fork --pid -- chroot "${TMP_DIR}" eatmydata apt-get --assume-yes --option=Dpkg::options::=--force-unsafe-io --option=Dpkg::Options::=--force-confold install linux-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-generic : Depends: linux-image-generic (= 5.4.0.186.184) but 5.4.0.177.175 is to be installed
                 Depends: linux-headers-generic (= 5.4.0.186.184) but 5.4.0.177.175 is to be installed
E: Unable to correct problems, you have held broken packages.
root@localhost:~# unshare  --fork --pid -- chroot "${TMP_DIR}" dpkg -l | grep linux-generic
hi  linux-generic                         5.4.0.177.175                     amd64        Complete Generic Linux kernel and headers
root@localhost:~#

A deploy using a freshly created custom image works because the custom image now has the latest kernel. But we don’t want to use new images unless necessary. What should be the recommended approach here? Appreciate any lead here.

Thank you!

trsoumi88 · 24 June 2024 15:32

Further debugging showed that, package ‘linux-generic’ has been set as hold

root@localhost:~# unshare  --fork --pid -- chroot "${TMP_DIR}" apt-cache show linux-generic=5.4.0.177.175
Package: linux-generic
Status: hold ok installed      <<<------------ 
Priority: optional
Section: kernel
Installed-Size: 20
Maintainer: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>
Architecture: amd64
Source: linux-meta
Version: 5.4.0.177.175
Provides: kernel-testing--linux--full--generic, kernel-testing--linux--full--preferred
Depends: linux-image-generic (= 5.4.0.177.175), linux-headers-generic (= 5.4.0.177.175)
Description-en: Complete Generic Linux kernel and headers
 This package will always depend on the latest complete generic Linux kernel
 and headers.
Description-md5: 000d0a6187a93215f75bba542cc6df27

root@localhost:~#

This looks to be the reason for package installation failure. Is there any way I can avoid installing kernel in the disk during the deploy stage?

thanks!

r00ta · 26 June 2024 14:15

could you elaborate more what you mean?

trsoumi88 · 27 June 2024 01:10

@r00ta, Thank you for looking into the issue.

During the deploy stage, I believe after the ephemeral image is loaded to memory, curtin (as part of the ephemeral image) starts executing a handful of tasks (as per a predefined set of stages. One of the stages in the process is stage-curthooks. As per the machine installation logs for a machine from MaaS, below is the sequence of steps that happen in stage-curthooks:

curtin: Installation started. (22.1-0ubuntu1~20.04.1)
.....
<logs redracted>
......
start: cmd-install/stage-curthooks/builtin/cmd-curthooks: curtin command curthooks

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-apt-config: configuring apt configuring apt
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-apt-config: SUCCESS: configuring apt configuring apt

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-missing-packages: installing missing packages
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-missing-packages: SUCCESS: installing missing packages

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-iscsi-service: configuring iscsi service
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-iscsi-service: SUCCESS: configuring iscsi service

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-mdadm-service: configuring raid (mdadm) service
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-mdadm-service: SUCCESS: configuring raid (mdadm) service

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: installing kernel
Running in chroot, ignoring request: start
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: SUCCESS: installing kernel

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/setting-up-swap: setting up swap
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/setting-up-swap: SUCCESS: setting up swap

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/apply-networking-config: apply networking config
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/apply-networking-config: SUCCESS: apply networking config

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: writing etc/fstab
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-etc-fstab: SUCCESS: writing etc/fstab

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-multipath: configuring multipath
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-multipath: SUCCESS: configuring multipath

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/system-upgrade: updating packages on target system
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/system-upgrade: SUCCESS: updating packages on target system

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/pollinate-user-agent: configuring pollinate user-agent on target
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/pollinate-user-agent: SUCCESS: configuring pollinate user-agent on target

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/updating-initramfs-configuration: updating initramfs configuration
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/updating-initramfs-configuration: SUCCESS: updating initramfs configuration

start: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-bootloader: configuring target system bootloader
start: cmd-install/stage-curthooks/builtin/cmd-curthooks/install-grub: installing grub to target devices
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/install-grub: SUCCESS: installing grub to target devices
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-bootloader: SUCCESS: configuring target system bootloader

finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: SUCCESS: curtin command curthooks

start: cmd-install/stage-hook/builtin/cmd-hook: curtin command hook
finish: cmd-install/stage-hook/builtin/cmd-hook: SUCCESS: curtin command hook

start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: curtin command in-target
finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init/cmd-in-target: SUCCESS: curtin command in-target

start: cmd-install/stage-late/99-validate-custom-image-has-netplan/cmd-in-target: curtin command in-target
finish: cmd-install/stage-late/99-validate-custom-image-has-netplan/cmd-in-target: SUCCESS: curtin command in-target

curtin: Installation finished.

There is an attempt to update-kernel on the target, which I am trying to avoid. The custom image has a status of hold for all the installed packages because my usecase needs it to be so.

I do understand that there are a handful of steps in the stage that we cannot avoid like updating-initramfs-configuration, configuring-bootloader, install-grub, but others can be avoided. Is there an easy way I can achieve this?

thanks!

r00ta · 27 June 2024 07:03

You might try to write your own curtin hooks for your custom image by patching the original and removing what you don’t need