Debugging initramfs used for network booting

Development
1

When booting a machine over the network, the machine first downloads the bootloader, which in turn downloads the kernel and initramfs. After the kernel is executed, it extracts and mount the initramfs as a rootfs in RAM. The initramfs is then responsible for mounting the real root filesystem and run the init (most likely systemd) process from there.

To understand exactly what happens when a machine is booting over the network, it can be useful to take a look at what the initramfs actually does, and even modify it a bit, for example to write debug information to the console.

Location of the initramfs file

You’ll find the kernel, initramfs and images on disk at /var/snap/maas/common/boot-resources/current for snap installs, and /var/lib/maas/boot-resources/current for deb installs.

In that directly, you’ll find the initramfs file you’re looking for at $distro_name/$arch/$kernel/$distro_series/$stream/boot-initrd.

For example, the initramfs for Ubuntu 22.04 for amd64 from the default stream using the generic kernel is here:

ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd

If you don’t have shell access to the machine where MAAS is running, you can also download it from the rackd server:

http://localhost:5248/images/ubuntu/amd64/ga-20.04/focal/stable/boot-initrd

Note that even though the filename is boot-initrd, it’s actually an initramfs, not an initrd.

Looking at what the initramfs contains

Now that you have the initramfs file as boot-initrd, you can look at what it contains using either lsinitramfs or unmkinitramfs. The former will list all the files in the initramfs archive, while the latter will unpack all the files into a directory.

For example, if you unpack the Ubuntu 22.04 initramfs it should look like this:

~$ unmkinitramfs boot-initrd myinit
~$ ls myinit
early  early2  main

The main directory is the most interesting one. It’s the directory that gets mounted as the rootfs by the kernel. The early and early2 directories contain CPU microcode that you most likely won’t have to bother about.

After the kernel has mounted the rootfs, it executes /init. That means that main/init is the file to look at to see what is happening. In essence, it will mount the real root filesystem and then execute /sbin/init from there.

There should be a line like this in there:

[    8.451583] Run /init as init process

That’s when init from the initramfs gets executed.

The role of initramfs for MAAS

Very simplified, for MAAS the initramfs does:

  • Configure networking
  • Download image
  • Mount image as '/`
  • Run /sbin/init

It then continues booting the image that it downloaded. Different things will happen, depending on whether the machine is commissioning, deploying, deploying disklessly, etc. Exactly what will happen in each case is controlled by cloud-init. MAAS configure itself as a cloud-init data source using kernel command line parameters.

Observing initramfs while it boots

If you add debug output to the initramfs, you can look at the output while booting a machine. For example, if you use a LXD VM, you can view the output while it’s booting using lxc console $vm_name

Modifying the initramfs

Sometimes it can be useful to make changes to the initramfs that is used when booting a machine over the network. For example, to add another kernel module, or insert debug print statements to see what is actually happening in more detail.

Note, this should be done on development deployments only, never in a production deployment.

After you’ve done the changes to the myinit/main directory, you need to rebuild the initramfs. It’s not a single archive, so you have to do several steps to get it back to boot-initrd. Assuming you’re in the myinit directory, you can run this:

cd early
find . -print0 | cpio --null --create --format=newc > ../new-boot-initrd
cd ../early2
find . -print0 | cpio --null --create --format=newc >> ../new-boot-initrd
cd ../main
find . | cpio --create --format=newc | xz --format=lzma >> ../new-boot-initrd

You can now copy the file back to where you got it. For example:

cp new-boot-initrd /var/snap/maas/common/maas/boot-resources/current/ubuntu/amd64/ga-22.04/jammy/stable/boot-initrd

Now if you deploy Ubuntu 22.04 for amd64 using the generic kernel, you’re customised initramfs should be used.

Note that if there are new images available, MAAS might overwrite you’re boot-initrd. When testing things it’s worth keeping an eye out to see that you’re initramfs is still there.

1 Like