Image deploy failure with Ubuntu 24.04

Hello! I have a snap-installed Maas 3.7.2 environment in which Ubuntu 24.04 deployments fail starting approximately Thursday May 12, 2026 at 5pm EDT. I also have another snap-installed MaaS 3.7.2 environment in which Ubuntu 24.04 deployments continue to correctly function. I’m rather at a loss as to the differences between these two environments, and what I need to change in the problematic environment so as to allow deployments of Ubuntu 24.04 to succeed. Based on the lack of questions I’ve seen regarding this type of error, I’m of the mindset that the issue is rather local to my environment in which it’s occurring and not a more widespread issue.

In the failing environment, I can see the from the logs that curtin successfully updates the repository metadata and then downloads linux-generic and its dependencies:

Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpikc6zv7n/target', 'apt-get', '--quiet', '--option=Acquire::Languages=none', '--option=Dir::Etc::sourcelist=/tmp/tmpizvkblnq/sources.list', '--option=Dir::Etc::sourceparts=/tmp/tmpizvkblnq/sources.list.d', 'update'] with allowed return codes [0] (capture=False)
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Get:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [269 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Components [35.0 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 c-n-f Metadata [8328 B]
Get:8 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [15.0 MB]
Get:9 http://archive.ubuntu.com/ubuntu noble/universe amd64 Components [3871 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/universe amd64 c-n-f Metadata [301 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [44.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:13 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 c-n-f Metadata [656 B]
Get:14 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1969 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [178 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [17.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [3124 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1689 kB]
Get:19 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [386 kB]
Get:20 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [34.5 kB]
Get:21 http://archive.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages [28.8 kB]
Get:22 http://archive.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Get:23 http://archive.ubuntu.com/ubuntu noble-security/multiverse amd64 c-n-f Metadata [396 B]
Get:24 http://archive.ubuntu.com/ubuntu noble-security/main amd64 Packages [1668 kB]
Get:25 http://archive.ubuntu.com/ubuntu noble-security/main amd64 Components [21.9 kB]
Get:26 http://archive.ubuntu.com/ubuntu noble-security/main amd64 c-n-f Metadata [11.0 kB]
Get:27 http://archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2943 kB]
Get:28 http://archive.ubuntu.com/ubuntu noble-security/restricted amd64 Components [212 B]
Get:29 http://archive.ubuntu.com/ubuntu noble-security/restricted amd64 c-n-f Metadata [544 B]
Get:30 http://archive.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1187 kB]
Get:31 http://archive.ubuntu.com/ubuntu noble-security/universe amd64 Components [74.1 kB]
Get:32 http://archive.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [23.1 kB]
Get:33 http://archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Packages [748 B]
Get:34 http://archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:35 http://archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 c-n-f Metadata [116 B]
Get:36 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Packages [40.6 kB]
Get:37 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [5776 B]
Get:38 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 c-n-f Metadata [368 B]
Get:39 http://archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [212 B]
Get:40 http://archive.ubuntu.com/ubuntu noble-backports/restricted amd64 c-n-f Metadata [116 B]
Get:41 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [31.0 kB]
Get:42 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.6 kB]
Get:43 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 c-n-f Metadata [1484 B]
Fetched 33.4 MB in 3s (13.0 MB/s)
Reading package lists...
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.001
Running command ['mount', '--make-private', '/tmp/tmpikc6zv7n/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpikc6zv7n/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpikc6zv7n/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpikc6zv7n/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpikc6zv7n/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpikc6zv7n/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpikc6zv7n/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpikc6zv7n/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/dev', '/tmp/tmpikc6zv7n/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/proc', '/tmp/tmpikc6zv7n/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/run', '/tmp/tmpikc6zv7n/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys', '/tmp/tmpikc6zv7n/target/sys'] with allowed return codes [0] (capture=False)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpikc6zv7n/target', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', '--download-only', 'linux-generic'] with allowed return codes [0] (capture=False)
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  amd64-microcode bpfcc-tools bpftrace firmware-sof-signed fontconfig-config
  fonts-dejavu-core fonts-dejavu-mono grub-common grub-gfxpayload-lists
  grub-pc grub-pc-bin grub2-common hwdata ieee-data intel-microcode
  iucode-tool libaom3 libbpfcc libc-dev-bin libc-devtools libc6-dev
  libclang-cpp18 libclang1-18 libcrypt-dev libdbus-glib-1-2 libdeflate0
  libefiboot1t64 libefivar1t64 libfontconfig1 libfreetype6 libgd3
  libheif-plugin-aomdec libheif-plugin-aomenc libheif1 libimobiledevice6
  libjbig0 libjpeg-turbo8 libjpeg8 liblerc4 libllvm18 libplist-2.0-4
  libsharpyuv0 libtiff6 libupower-glib3 libusbmuxd6 libwebp7 libxpm4
  linux-firmware linux-headers-6.8.0-111 linux-headers-6.8.0-111-generic
  linux-headers-generic linux-image-6.8.0-111-generic linux-image-generic
  linux-libc-dev linux-modules-6.8.0-111-generic
  linux-modules-extra-6.8.0-111-generic linux-tools-6.8.0-111
  linux-tools-6.8.0-111-generic linux-tools-common manpages-dev os-prober
  python3-bpfcc python3-netaddr rpcsvc-proto thermald
  ubuntu-kernel-accessories upower usbmuxd wireless-regdb
Suggested packages:
  multiboot-doc grub-emu mtools xorriso desktop-base glibc-doc libgd-tools
  libheif-plugin-libde265 libheif-plugin-x265 libheif-plugin-ffmpegdec
  libheif-plugin-jpegdec libheif-plugin-jpegenc libheif-plugin-j2kdec
  libheif-plugin-j2kenc libheif-plugin-rav1e libheif-plugin-svtenc
  libusbmuxd-tools fdutils linux-tools ipython3 python-netaddr-docs
The following NEW packages will be installed:
  amd64-microcode bpfcc-tools bpftrace firmware-sof-signed fontconfig-config
  fonts-dejavu-core fonts-dejavu-mono grub-common grub-gfxpayload-lists
  grub-pc grub-pc-bin grub2-common hwdata ieee-data intel-microcode
  iucode-tool libaom3 libbpfcc libc-dev-bin libc-devtools libc6-dev
  libclang-cpp18 libclang1-18 libcrypt-dev libdbus-glib-1-2 libdeflate0
  libefiboot1t64 libefivar1t64 libfontconfig1 libfreetype6 libgd3
  libheif-plugin-aomdec libheif-plugin-aomenc libheif1 libimobiledevice6
  libjbig0 libjpeg-turbo8 libjpeg8 liblerc4 libllvm18 libplist-2.0-4
  libsharpyuv0 libtiff6 libupower-glib3 libusbmuxd6 libwebp7 libxpm4
  linux-firmware linux-generic linux-headers-6.8.0-111
  linux-headers-6.8.0-111-generic linux-headers-generic
  linux-image-6.8.0-111-generic linux-image-generic linux-libc-dev
  linux-modules-6.8.0-111-generic linux-modules-extra-6.8.0-111-generic
  linux-tools-6.8.0-111 linux-tools-6.8.0-111-generic linux-tools-common
  manpages-dev os-prober python3-bpfcc python3-netaddr rpcsvc-proto thermald
  ubuntu-kernel-accessories upower usbmuxd wireless-regdb
0 upgraded, 70 newly installed, 0 to remove and 14 not upgraded.
Need to get 922 MB of archives.
After this operation, 1286 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libllvm18 amd64 1:18.1.3-1ubuntu1 [27.5 MB]
Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libclang-cpp18 amd64 1:18.1.3-1ubuntu1 [13.5 MB]
Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 libbpfcc amd64 0.29.1+ds-1ubuntu7 [663 kB]
Get:4 http://archive.ubuntu.com/ubuntu noble/main amd64 python3-bpfcc all 0.29.1+ds-1ubuntu7 [40.2 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble/main amd64 ieee-data all 20220827.1 [2113 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 python3-netaddr all 0.8.0-2ubuntu1 [319 kB]
Get:7 http://archive.ubuntu.com/ubuntu noble/main amd64 bpfcc-tools all 0.29.1+ds-1ubuntu7 [687 kB]
Get:8 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libclang1-18 amd64 1:18.1.3-1ubuntu1 [7803 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 bpftrace amd64 0.20.2-1ubuntu4.3 [1011 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 firmware-sof-signed all 2023.12.1-1ubuntu1.10 [7382 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble/main amd64 fonts-dejavu-mono all 2.37-8 [502 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble/main amd64 fonts-dejavu-core all 2.37-8 [835 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble/main amd64 fontconfig-config amd64 2.15.0-1.1ubuntu2 [37.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble/main amd64 libefivar1t64 amd64 38-3.1build1 [57.8 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble/main amd64 libefiboot1t64 amd64 38-3.1build1 [40.6 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libfreetype6 amd64 2.13.2+dfsg-1ubuntu0.1 [402 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 grub-common amd64 2.12-1ubuntu7.3 [2120 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 grub2-common amd64 2.12-1ubuntu7.3 [670 kB]
Get:19 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 grub-pc-bin amd64 2.12-1ubuntu7.3 [1101 kB]
Get:20 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 grub-pc amd64 2.12-1ubuntu7.3 [137 kB]
Get:21 http://archive.ubuntu.com/ubuntu noble/main amd64 grub-gfxpayload-lists amd64 0.7build2 [3518 B]
Get:22 http://archive.ubuntu.com/ubuntu noble/main amd64 hwdata all 0.379-1 [29.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu noble/main amd64 iucode-tool amd64 2.3.1-3build1 [43.9 kB]
Get:24 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libaom3 amd64 3.8.2-2ubuntu0.1 [1941 kB]
Get:25 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc-dev-bin amd64 2.39-0ubuntu8.7 [20.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu noble/main amd64 libfontconfig1 amd64 2.15.0-1.1ubuntu2 [139 kB]
Get:27 http://archive.ubuntu.com/ubuntu noble/main amd64 libsharpyuv0 amd64 1.3.2-0.4build3 [15.8 kB]
Get:28 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libheif-plugin-aomdec amd64 1.17.6-1ubuntu4.3 [10.6 kB]
Get:29 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libheif1 amd64 1.17.6-1ubuntu4.3 [276 kB]
Get:30 http://archive.ubuntu.com/ubuntu noble/main amd64 libjpeg-turbo8 amd64 2.1.5-2ubuntu2 [150 kB]
Get:31 http://archive.ubuntu.com/ubuntu noble/main amd64 libjpeg8 amd64 8c-2ubuntu11 [2148 B]
Get:32 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libdeflate0 amd64 1.19-1build1.1 [43.9 kB]
Get:33 http://archive.ubuntu.com/ubuntu noble/main amd64 libjbig0 amd64 2.1-6.1ubuntu2 [29.7 kB]
Get:34 http://archive.ubuntu.com/ubuntu noble/main amd64 liblerc4 amd64 4.0.0+ds-4ubuntu2 [179 kB]
Get:35 http://archive.ubuntu.com/ubuntu noble/main amd64 libwebp7 amd64 1.3.2-0.4build3 [230 kB]
Get:36 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libtiff6 amd64 4.5.1+git230720-4ubuntu2.5 [200 kB]
Get:37 http://archive.ubuntu.com/ubuntu noble/main amd64 libxpm4 amd64 1:3.5.17-1build2 [36.5 kB]
Get:38 http://archive.ubuntu.com/ubuntu noble/main amd64 libgd3 amd64 2.3.3-9ubuntu5 [128 kB]
Get:39 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc-devtools amd64 2.39-0ubuntu8.7 [29.3 kB]
Get:40 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-libc-dev amd64 6.8.0-111.111 [1586 kB]
Get:41 http://archive.ubuntu.com/ubuntu noble/main amd64 libcrypt-dev amd64 1:4.4.36-4build1 [112 kB]
Get:42 http://archive.ubuntu.com/ubuntu noble/main amd64 rpcsvc-proto amd64 1.4.2-0ubuntu7 [67.4 kB]
Get:43 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-dev amd64 2.39-0ubuntu8.7 [2124 kB]
Get:44 http://archive.ubuntu.com/ubuntu noble/main amd64 libdbus-glib-1-2 amd64 0.112-3build2 [64.2 kB]
Get:45 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libheif-plugin-aomenc amd64 1.17.6-1ubuntu4.3 [14.7 kB]
Get:46 http://archive.ubuntu.com/ubuntu noble/main amd64 libplist-2.0-4 amd64 2.3.0-1~exp2build2 [48.0 kB]
Get:47 http://archive.ubuntu.com/ubuntu noble/main amd64 libusbmuxd6 amd64 2.0.2-4build3 [20.3 kB]
Get:48 http://archive.ubuntu.com/ubuntu noble/main amd64 libimobiledevice6 amd64 1.3.0-8.1build3 [70.3 kB]
Get:49 http://archive.ubuntu.com/ubuntu noble/main amd64 libupower-glib3 amd64 1.90.3-1 [39.0 kB]
Get:50 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-firmware amd64 20240318.git3b128b60-0ubuntu2.27 [641 MB]
Get:51 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-modules-6.8.0-111-generic amd64 6.8.0-111.111 [39.2 MB]
Get:52 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-image-6.8.0-111-generic amd64 6.8.0-111.111 [14.8 MB]
Get:53 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 wireless-regdb all 2025.10.07-0ubuntu1~24.04.1 [7542 B]
Get:54 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-modules-extra-6.8.0-111-generic amd64 6.8.0-111.111 [113 MB]
Get:55 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 intel-microcode amd64 3.20260210.0ubuntu0.24.04.1 [13.5 MB]
Get:56 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 amd64-microcode amd64 3.20250311.1ubuntu0.24.04.1 [290 kB]
Get:57 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-image-generic amd64 6.8.0-111.111 [11.2 kB]
Get:58 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-headers-6.8.0-111 all 6.8.0-111.111 [13.7 MB]
Get:59 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-headers-6.8.0-111-generic amd64 6.8.0-111.111 [3905 kB]
Get:60 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-headers-generic amd64 6.8.0-111.111 [11.1 kB]
Get:61 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-generic amd64 6.8.0-111.111 [1692 B]
Get:62 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-tools-common all 6.8.0-111.111 [426 kB]
Get:63 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-tools-6.8.0-111 amd64 6.8.0-111.111 [5091 kB]
Get:64 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 linux-tools-6.8.0-111-generic amd64 6.8.0-111.111 [1818 B]
Get:65 http://archive.ubuntu.com/ubuntu noble/main amd64 manpages-dev all 6.7-2 [2013 kB]
Get:66 http://archive.ubuntu.com/ubuntu noble/main amd64 os-prober amd64 1.81ubuntu4 [18.0 kB]
Get:67 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 thermald amd64 2.5.6-2ubuntu0.24.04.5 [214 kB]
Get:68 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 ubuntu-kernel-accessories amd64 1.539.2 [10.9 kB]
Get:69 http://archive.ubuntu.com/ubuntu noble/main amd64 upower amd64 1.90.3-1 [84.4 kB]
Get:70 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 usbmuxd amd64 1.1.1-5~exp3ubuntu2.1 [42.3 kB]
Fetched 922 MB in 3s (309 MB/s)
Download complete and in download only mode

After the packages are downloaded, curtin successfully installs them.

In the problematic environment, however, curtin issues the apt-get update command to update repository metadata, but nothing is downloaded, and the subsequent command to download linux-generic and its dependencies fails because the package cannot be found:

Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpun4nl9ha/target', 'apt-get', '--quiet', '--option=Acquire::Languages=none', '--option=Dir::Etc::sourcelist=/tmp/tmp5x1qkmpd/sources.list', '--option=Dir::Etc::sourceparts=/tmp/tmp5x1qkmpd/sources.list.d', 'update'] with allowed return codes [0] (capture=False)
Reading package lists...
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.013
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/dev', '/tmp/tmpun4nl9ha/target/dev'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/proc', '/tmp/tmpun4nl9ha/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/run', '/tmp/tmpun4nl9ha/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--bind', '/sys', '/tmp/tmpun4nl9ha/target/sys'] with allowed return codes [0] (capture=False)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpun4nl9ha/target', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', '--download-only', 'linux-generic'] with allowed return codes [0] (capture=False)
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-generic
Running command ['udevadm', 'settle'] with allowed return codes [0] (capture=False)
TIMED subp(['udevadm', 'settle']): 0.013
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/sys'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/sys'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/run'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/run'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/proc'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/proc'] with allowed return codes [0] (capture=False)
Running command ['mount', '--make-private', '/tmp/tmpun4nl9ha/target/dev'] with allowed return codes [0] (capture=False)
Running command ['umount', '/tmp/tmpun4nl9ha/target/dev'] with allowed return codes [0] (capture=False)
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: FAIL: installing kernel
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
Traceback (most recent call last):
  File "/curtin/curtin/commands/main.py", line 202, in main
    ret = args.func(args)
          ^^^^^^^^^^^^^^^
  File "/curtin/curtin/commands/curthooks.py", line 1947, in curthooks
    builtin_curthooks(cfg, target, state)
  File "/curtin/curtin/commands/curthooks.py", line 1779, in builtin_curthooks
    install_kernel(cfg, target)
  File "/curtin/curtin/commands/curthooks.py", line 387, in install_kernel
    install(kernel_package)
  File "/curtin/curtin/commands/curthooks.py", line 364, in install
    distro.install_packages([pkg], target=target, env=env)
  File "/curtin/curtin/distro.py", line 526, in install_packages
    return install_cmd('install', args=pkglist, opts=opts, target=target,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/curtin/curtin/distro.py", line 308, in run_apt_command
    cmd_rv = apt_install(mode, args, opts=opts, env=env, target=target,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/curtin/curtin/distro.py", line 356, in apt_install
    cmd_rv = inchroot.subp(cmd + dl_opts + packages, env=env,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/curtin/curtin/util.py", line 792, in subp
    return subp(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/curtin/curtin/util.py", line 280, in subp
    return _subp(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/curtin/curtin/util.py", line 144, in _subp
    raise ProcessExecutionError(stdout=out, stderr=err,
curtin.util.ProcessExecutionError: Unexpected error while running command.

The only difference I can find in these environments is the version of curtin being used.

Version of curtin from the environment in which Ubuntu 24.04 deploys succeed:

curtin: Installation started. (23.1.1-1118-g3977ce90-0ubuntu1~ubuntu24.04.1)

Version of curtin from the environment in which Ubuntu 24.04 deploys fail:

curtin: Installation started. (23.1.1-1124-g7324b43b-0ubuntu1~ubuntu24.04.1)

I’ve tried removing and re-adding the 24.04 boot source with the thought that perhaps the environment is using a corrupted version of the image, but based on the speed with which the image is synced after re-adding the 24.04 boot source I am doubtful that removing the source actually removes the image from the controller.

This issue is having quite the impact on my team’s use of our MaaS environment, and it has me very puzzled. Any thoughts, suggestions, or advice would be greatly appreciated, and thank you in advance for your time.

A few additional notes:

  • Deployments of Ubuntu 22.04 do not appear to be affected by this issue.
  • I used the preseed template to inject a timedatectl command to verify that the date and time are correct on the machines being deployed in each environment.
  • I tried disabling the MaaS proxy to test the theory that something may have gone awry with the caching proxy, but doing so resulted in no change in behavior.
  • I learned how to find the name of the squashfs file associated with an image, and confirmed that the image file on disk is indeed not corrupt and in fact is identical between the behaving and misbehaving environments.

Probably the archive was not properly working. If you want to have a stable environment that is not subject to the uptime of the archive you can create your own mirror of apt

A local mirror is definitely something to consider for a long-term solution, as it should also speed up deployments.

I would agree that the archive was probably not working, but the error has been happening with every single 24.04 install in this environment for a couple of days now. I would expect that the probability I’m hitting one particular broken mirror every single time over the course of several days in one environment, and avoiding that broken mirror every single time over the course of several days in another environment, would be rather low. Would you agree that this issue seems too consistent to be indicative of a single broken mirror, or even a small set of them?

How is it possible that you are using

curtin: Installation started. (23.1.1-1124-g7324b43b-0ubuntu1~ubuntu24.04.1)

In your 3.7/stable environment? That version of curtin was never part of a stable release. We detected that it was broken during certification and we never landed it into the stable release.

Aha - I believe I understand. The snap that was installed in this environment was build from the MaaS sources after applying some patches that are necessary for this environment. In looking at the manifest files, I see there are certainly some differences in the packages that are included in the snap - notably the python3-curtin version. So this issue is unfortunately self-inflicted. Thank you for your help in pointing out the flaw in our process!

I attempted to build the 3.7.2 snap from the tag in the MaaS repository, but I see from the manifest that I’m now getting curtin-common and python3-curtin version 23.1.1-1127-g68310ce3-0ubuntu1~ubuntu24.04.1. Is there a way to reproduce the components included in the stable release of the MaaS snap?

Actually I don’t know what you are doing when you patched the codebase and rebuilt the snap. In theory if you did things right you should end up using the same curtin version that is part of the stable release. BTW if you have the chance to test the version 1127 and report back if it works it’s even better

I unfortunately haven’t had a chance to take 1127 for a test drive, but I will happily report back as soon as I do.

The changes are fairly minimal: minor tweaks to disable X-Forwarded-Proto because of HAProxy and TLS issues; modification to the regiond.nginx.conf.template file so that Prometheus metrics are available to only the localhost. To build the snap I run:

make snap-clean ; make clean ; git clean -dfx ; time make snap

I thought I got the make snap from the README files, but I’m having difficulty finding which one so I’m very prepared to be incorrectly recalling…

Did you checkout the 3.7 branch? I’d suggest to spin up a new container and start from scratch.