How to build MAAS images

MAAS can deploy custom OS images. Canonical provides both lp:maas-image-builder^ and gh:canonical/packer-maas^ to support creating custom images. MAAS Image Builder requires the purchase of Ubuntu Pro support.

Pro tip: While it may be possible to deploy a certain image with MAAS, the particular use case may not be supported by that image’s vendor due to licensing or technical reasons. Canonical recommends that, whenever possible, you should customise machines using cloud-init user_data or Curtin preseed data, instead of creating a custom image.

Using MAAS version 3.1 and above, custom images can include static Ubuntu images, created with whatever tool you choose, as well as other OS images. For MAAS versions 3.0 and below, MAAS supports deploying custom DD or TGZ images. Customised Ubuntu deployments aren’t well supported for MAAS 3.0 and below.

The MAAS stream^ is generated by lp:maas-images^ as follows:

  1. It downloads the universal SquashFS rootfs (no kernel) from cloud-images.ubuntu.com and mounts it as an overlay.
  2. It then installs a kernel and scripts from cloud-initramfs-rooturl and cloud-initramfs-copymods to allow network booting.
  3. The kernel and initramfs are pulled out of the overlay and everything is unmounted.
  4. lp:maas-images^ provides the unmodified SquashFS, installed kernel, and generated initramfs as separate files on images.maas.io.

MAAS uses this package to network boot for commissioning, testing, and deployment.

There are two methods for building custom images to be deployed to MAAS machines: MAAS Image Builder, and packer^.

About older MAAS versions

Older versions of MAAS only allow a single TGZ or DD.GZ (no separate files). Custom images are always deployed with the ephemeral operating system, so some debugging may be required. Here is a procedure:

  • Download the rootfs from images.maas.io
    wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-root.tar.xz

  • Create a work directory
    mkdir /tmp/work

  • Extract the rootfs
    cd /tmp/work
    sudo tar xf ../focal-server-cloudimg-amd64-root.tar.xz

    or

    unsquashfs ../focal-server-cloudimg-amd64-root.squash

  • Setup chroot
    sudo mount -o bind /proc /tmp/work/proc
    sudo mount -o bind /dev /tmp/work/dev
    sudo mount -o bind /sys /tmp/work/sys
    sudo mv /tmp/work/etc/resolv.conf /tmp/work/etc/resolv.conf.bak
    sudo cp /etc/resolv.conf /tmp/work/etc/

  • Chroot in
    sudo chroot /tmp/work /bin/bash

  • Customise image
    apt update
    apt dist-upgrade
    apt install emacs-nox jq tree
    apt autoremove

  • Exit chroot and unmount binds
    exit
    sudo umount /tmp/work/proc
    sudo umount /tmp/work/dev
    sudo umount /tmp/work/sys
    sudo mv /tmp/work/etc/resolv.conf.bak /tmp/work/etc/resolv.conf

  • Create TGZ
    sudo tar -czf ~/focal-custom.tgz -C /tmp/work .

  • Upload it to MAAS
    Note: Ubuntu release names and versions are reserved
    maas $PROFILE boot-resources create name='custom/focal-custom' title='Ubuntu 20.04 Custom Image' architecture='amd64/generic' filetype='tgz' content@=~/focal-custom.tgz

  • Configure and deploy as normal

Verify prerequisites

The following are required to to create a packer MAAS image:

  • A machine running Ubuntu 18.04+ or higher with the ability to run KVM virtual machines.
  • Various dependencies required by the chosen template (see the template directory for details)
  • Packer^

As an example for this article, we will be building a custom Ubuntu image from the default template, which has the following prerequisites:

  • qemu-utils
  • qemu-system
  • ovmf
  • cloud-image-utils

Note that these requirements may vary by template and target image.

Verify deploy requirements

The following are required to deploy a packer MAAS image:

Customise images

It is possible to customize the image either during the Ubuntu installation, or afterwards (before packing the final image). The former is done by providing autoinstall config^, editing the user-data-flat and user-data-lvm files. The latter is performed by the install-custom-packages script.

Build by proxy

The Packer template downloads the Ubuntu net installer from the Internet. To tell Packer to use a proxy, set the HTTP_PROXY environment variable to your proxy server. Alternatively, you may redefine iso_url to a local file, set iso_checksum_type to none to disable the checksum, and remove iso_checksum_url.

Build checklist

Packer^ can be used to build images to be deployed by MAAS. In order to use packer, you must have a packer template^ for the OS version you intend to build.

Currently, templates are available for:

  • Ubuntu custom images
  • CentOS6
  • CentOS7
  • CentOS8
  • RHEL7
  • RHEL8
  • VMWare ESXi

Note that additional templates will be made available from time to time.

Install packer

Packer is easily installed from its Debian package:

sudo apt install packer

For this example Ubuntu template, the following dependencies should also be installed – but note that these dependencies may vary by template and/or target image:

sudo apt install qemu-utils
sudo apt install qemu-system
sudo apt install ovmf
sudo apt install cloud-image-utils

All of these should install without additional prompts.

Obtain templates

You can obtain the packer templates by cloning the packer-maas github repository^, like this:

git clone https://github.com/canonical/packer-maas.git

Make sure to pay attention to where the repository is cloned. The Packer template in this cloned repository creates a Ubuntu AMD64 image for use with MAAS.

Create images

To build a packer image, you must change to the template repository directory, then to the subdirectory for the image you want to build. From that subdirectory, you can easily build a raw image with LVM, using the Makefile:

$ make custom-ubuntu-lvm.dd.gz

This makefile will run for a couple of minutes before attempting to boot the image. While waiting for the image to boot, packer will attempt to SSH into the machine repeatedly until it succeeds. You will see terminal messages similar to this one for upwards of three to five minutes:

2022/05/09 15:50:46 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2022/05/09 15:50:50 packer-builder-qemu plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
2022/05/09 15:50:50 packer-builder-qemu plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.

Eventually, you should see a successful SSH connection:

2022/05/09 15:50:57 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2351...
2022/05/09 15:50:57 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/05/09 15:50:57 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2022/05/09 15:51:17 packer-builder-qemu plugin: [DEBUG] handshake complete!

If the process seems to run for a long time, you can predict whether it’s going to work by doing a series of netstat -a on the IP:port given in the connection attempt. Attempts may fail repeatedly, but if you repeat the netstat -a command frequently, you will see some tentative connections, like this one:

stormrider@neuromancer:~$ netstat -a | grep 2281
tcp        0      0 0.0.0.0:2281            0.0.0.0:*               LISTEN     
tcp        0      0 localhost:46142         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46120         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46138         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46134         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46130         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46124         localhost:2281          TIME_WAIT  
stormrider@neuromancer:~$ netstat -a | grep 2281
tcp        0      0 0.0.0.0:2281            0.0.0.0:*               LISTEN     
tcp        0      0 localhost:46142         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46146         localhost:2281          ESTABLISHED
tcp        0      0 localhost:2281          localhost:46146         ESTABLISHED
tcp        0      0 localhost:46138         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46134         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46130         localhost:2281          TIME_WAIT  
tcp        0      0 localhost:46124         localhost:2281          TIME_WAIT

This ESTABLISHED connection may not hold the first few times, but eventually, the SSH connection will be made, and the provisioning process will finish. If you want to walk away and come back, be advised that the Makefile clears the terminal buffer at the end, but echoes one final instruction:

rm OVMF_VARS.fd

You can check the validity of the operation with ls, like this:

stormrider@neuromancer:~/mnt/Dropbox/src/git/packer-maas/ubuntu$ ls
custom-ubuntu-lvm.dd.gz  packages      seeds-lvm.iso     user-data-lvm
http                     packer_cache  ubuntu-flat.json
Makefile                 README.md     ubuntu-lvm.json
meta-data                scripts       user-data-flat

You can also manually run packer. Your current working directory must be in the subdirectory where the packer template is located. In the case of this example, that’s packer-maas/ubuntu. Once in packer-maas/ubuntu, you can generate an image with the following command sequence:

$ sudo PACKER_LOG=1 packer build ubuntu-lvm.json

ubuntu-lvm.json is configured to run Packer in headless mode. Only Packer output will be seen. If you wish to see the installation output connect to the VNC port given in the Packer output, or change the value of headless to false in the JSON file.

This process is non-interactive.

Upload to MAAS

You can upload a packer image with the following command:

$ maas admin boot-resources create \
    name='custom/ubuntu-raw' \
    title='Ubuntu Custom RAW' \
    architecture='amd64/generic' \
    filetype='ddgz' \
    content@=custom-ubuntu-lvm.dd.gz

Before relying on it in production, you should test your custom image by deploying it to a test (virtual) machine. It’s the machine named open-gannet in this listing:

maas admin machines read | jq -r '(["HOSTNAME","SYSID","POWER","STATUS",
"OWNER", "OS", "DISTRO"] | (., map(length*"-"))),
(.[] | [.hostname, .system_id, .power_state, .status_name, .owner // "-",
.osystem, .distro_series]) | @tsv' | column -t

HOSTNAME     SYSID   POWER  STATUS    OWNER  OS      DISTRO
--------     -----   -----  ------    -----  --      ------
valued-moth  e86c7h  on     Deployed  admin  ubuntu  focal
open-gannet  nk7x8y  on     Deployed  admin  custom  ubuntu-raw

Default username

The default username for packer-created images is ubuntu, the same as the default username for other MAAS images.