Creating a custom Ubuntu image

Docs
1

Introduction

MAAS supports deploying custom DD or TGZ images. Canonical provides both lp:maas-image-builder and gh:canonical/packer-maas to support creating custom images however these tools do not currently support Ubuntu. Instead Canonical suggests customizing Ubuntu using cloud-init user_data or Curtin preseed data.

Why customized Ubuntu deployments aren’t supported

When the MAAS stream is generated by lp:maas-images it starts by downloading the base SquashFS rootfs from cloud-images.ubuntu.com that is used for all clouds. The SquashFS does not contain a kernel so lp:maas-images mounts the SquashFS with an overlay and chroots in. It then installs a kernel and extra initramfs scripts from the cloud-initramfs-rooturl and cloud-initramfs-copymods packages to allow network booting. Once everything is installed the kernel and newly generated initramfs are pulled out of the overlay and everything is unmounted. lp:maas-images provides the unmodified SquashFS, installed kernel, and generated initramfs as separate files on images.maas.io.

MAAS uses the kernel, initramfs, and SquashFS to perform network booting which allows commissioning, testing, and deployments. When deploying Ubuntu MAAS uses the same version of Ubuntu to network boot into and perform the deployment. This ensures there are no compatibility issues. Other operating systems use the Ubuntu version selected for the ephemeral environment for deployment.

Currently MAAS only allows custom images to be loaded as a single TGZ or DD.GZ, there is no way to upload a kernel, initrd, and rootfs as separate files. Even if MAAS was modified to allow the kernel, initrd, and rootfs as separate files MAAS requires cloud-initramfs-copymods and cloud-initramfs-rooturl to be included in the initrd. It is difficult for MAAS to detect if these scripts are missing and even harder for users to debug if they are missing.

Warnings on creating a custom Ubuntu image

  1. Custom images are always deployed with the ephemeral operating system. This can cause hard to debug errors. For example CentOS 6 can only be deployed by Ubuntu Xenial due to advances in the ext4 filesystem.
  2. Custom Ubuntu images cannot be used for commissioning, testing, rescue mode, or any other ephemeral environment.
  3. Curtin will still install and configure the GA kernel and GRUB. If your custom image contains a kernel it most likely won’t be used. MAAS will not allow you to select which kernel(GA, HWE, lowlatency, etc) when deploying a custom Ubuntu image, you always get the latest GA kernel.
  4. Curtin will still update /etc/apt/sources.list with what is configured in MAAS.
  5. cloud-init will still reconfigure network and NTP settings.
  6. All GNU/Linux custom images should be created as a TGZ to enable storage customization. When a DD.GZ is used MAAS/Curtin simply writes the file to the filesystem.

How to create a custom Ubuntu image for MAAS

Note: LXD may prevent device files from being created when extracting the rootfs, it is suggested to do this on metal or on a VM

  1. Download the rootfs from images.maas.io
    wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-root.tar.xz

  2. Create a work directory
    mkdir /tmp/work

  3. Extract the rootfs
    cd /tmp/work
    sudo tar xf ../focal-server-cloudimg-amd64-root.tar.xz

    or

    unsquashfs ../focal-server-cloudimg-amd64-root.squash

  4. Setup chroot
    sudo mount -o bind /proc /tmp/work/proc
    sudo mount -o bind /dev /tmp/work/dev
    sudo mount -o bind /sys /tmp/work/sys
    sudo mv /tmp/work/etc/resolv.conf /tmp/work/etc/resolv.conf.bak
    sudo cp /etc/resolv.conf /tmp/work/etc/

  5. Chroot in
    sudo chroot /tmp/work /bin/bash

  6. Customize image
    apt update
    apt dist-upgrade
    apt install emacs-nox jq tree
    apt autoremove

  7. Exit chroot and unmount binds
    exit
    sudo umount /tmp/work/proc
    sudo umount /tmp/work/dev
    sudo umount /tmp/work/sys
    sudo mv /tmp/work/etc/resolv.conf.bak /tmp/work/etc/resolv.conf

  8. Create TGZ
    sudo tar -czf /tmp/focal-custom.tgz -C /tmp/work .

  9. Upload it to MAAS
    Note: Ubuntu release names and versions are reserved
    maas $PROFILE boot-resources create name='custom/focal-custom' title='Ubuntu 20.04 Custom Image' architecture='amd64/generic' filetype='tgz' content@=focal-custom.tgz

  10. Configure and deploy as normal

Using a custom/specific kernel with an Ubuntu custom image

As mentioned in warnings on creating a custom Ubuntu image section Curtin will always install the GA kernel. You can work around this by modifying the Curtin preseed for custom images to remove the GA kernel. curtin_userdata_custom should be modified as follows:

#cloud-config
debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  remove_kernel: ['curtin', 'in-target', '--', 'apt-get', 'purge', '-y', 'linux-image-generic', 'linux-*-generic']
  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']

Couple of caveats with this work around

  1. You can’t use the GA kernel in your custom image using this work around.
  2. You can’t select which kernel you wish to use in MAAS at deployment time.
  3. The Curtin preseed file must be updated on every MAAS region controller.
2 Likes