Raspberry Pi 4 provisioning and KVM pod setup

Hi, these days I have been trying to provision a raspberry pi 4 through MAAS and I did it. I did that using the Raspberry Pi 4 UEFI Firmware, but I had to fix a few problems.

Hardware

I used a raspberry pi 4 8 gb, a micro sd card for the UEFI firmware and an usb drive (where the os will be installed).

UEFI Firmware Setup

The Raspberry Pi 4 UEFI Firmware can perform a pxe boot, but currently (version 1.21) there are some problems that need to be fixed to work properly with MAAS:

  1. Versions after 1.16 do not work. Apparently, a commit related to version 1.17 broke the network: the uefi firmware can download the grub bootloader, but then It always shows the error message “couldn’t send network packet”. In grub most of the commands related to network (like net_dhcp) show that error. However, version 1.16 works, and it managed to download kernel files. I opened an issue.
  2. After the installation of ubuntu, maas sends to the client a different configuration file, that tells to scan every device to find the bootloader. However, when the rpi4 uefi firmware boots normally, it does not attach every device, but only the one from which was set to boot from. So, the first reboot after the deploy step fails, because the os bootloader cannot be found. It is possible to avoid this by pressing esc to enter the uefi setup and then press continue. By doing this all device are connected and the os can be booted, but you need to do it manually every time. I opened an issue.

I solved these problems recompiling the firmware, using the network drivers from version 1.16 and adding the line removed from that commit.

As for the uefi firmware configuration, I removed the 3 gb limit (Device Manager -> Raspberry Pi Configuration -> Advanced Configuration -> Limit RAM to 3 GB -> Disabled) and set the boot order to start from the network (Boot Maintenance Manager -> Boot Options -> Change Boot Order -> UEFI PXEv4 on top).

MAAS Configuration

I used snap MAAS 2.9/candidate.

In the MAAS UI, I selected to download ubuntu 20.04 and 20.10 for arm64 and I setup dhcp normally. I also set in the settings ubuntu 20.04 with the ga-20.04 minimum kernel as the default ubuntu release for commissioning and ubuntu 20.10 for deployment.

To work properly, the rpi4 with the UEFI firmware needs the 5.8 linux kernel (otherwise the network will not work, and the RAM must be limited to 3 GB). That kernel is present only in ubuntu groovy (20.10). MAAS can deploy ubuntu groovy, but the listing and commissioning steps can use, at most, ubuntu focal (that has linux kernel 5.4). The fastest solution that I found to solve this problem is to replace the ubuntu focal image files with the groovy ones. I used the mount and unmount commands to mount the groovy folder on top of the focal one, to quickly change images by mounting and unmounting folders.

I disabled the image autosync in MAAS and then replaced the image files:

sudo mount --bind -o nodev,ro /var/snap/maas/common/maas/boot-resources/snapshot–/ubuntu/arm64/ga-20.10/groovy/stable /var/snap/maas/common/maas/boot-resources/snapshot-/ubuntu/arm64/ga-20.04/focal/stable

After that, the raspberry can be powered on and the listing and commissioning step should succeed (using ubuntu 20.04 that in reality is ubuntu 20.10).
After the commissioning, the images can be reverted by unmounting the folder:

sudo umount /var/snap/maas/common/maas/boot-resources/snapshot-/ubuntu/arm64/ga-20.04/focal/stable

I do not know if there is a better way to commission a machine with linux kernel 5.8. A good solution would be to update the kernel of the 20.04 image, so that it would also possible to deploy ubuntu 20.04 directly on the raspberry, that is better supported (also by juju bundles that are all for focal and not for groovy). I do not know if that could be possible.

KVM Pod setup

After the commissioning, ubuntu 20.10 can be deployed.

Currently from the UI is not possible to deploy a node with ubuntu 20.10 as a kvm pod. That must be done from the cli:

maas $PROFILE machine deploy $SYSTEM_ID install_kvm=True

Since the default ubuntu release for the deployment was set to ubuntu 20.10, that release will be deployed.

After the deployment, the kvm pod will show up in the UI, but when trying to compose a machine il will throw an error. That is caused by the virsh template used by MAAS to compose the virtual machine for arm64, that apparently is not suitable for the rpi4.

The file /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod/virsh.py must be changed. However, that file is inside snap and cannot be changed. I solved this problem by mounting another folder with the same files on top of it.

I copied the folder with these files in my home directory:

sudo cp -r /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod ~/pod

Changed owner to my user to edit them:

sudo chown -R <you_user> ~/pod

Edited the file ~/pod/virsh.py. The template “DOM_TEMPLATE_ARM64” must be changed as follow:

  • Change <gic version='3'/> to <gic version='2'/>
  • Change <input type='mouse' bus='ps2'/> to <input type='mouse' bus='virtio'/>
  • Change <input type='keyboard' bus='ps2'/> to <input type='keyboard' bus='virtio'/>

Mount the folder on top of the snap one:

sudo mount --bind -o nodev,ro ~/pod /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod

Restart MAAS:

sudo snap restart maas.supervisor

The folder must be remounted every time the MAAS controller is rebooted if you want to compose a new vm.

After these steps it should be possible to compose some virtual machines on the rpi4, using any release of ubuntu arm64.

Deploy JUJU bundles

I tried to deploy the Kubernetes core bundle and it works. However, you must first create the virtual machines on maas and then deploy the juju bundle. I don’t know why, but if you don’t do that, juju will create the vm, but it will misconfigure the storage (not creating the fat32 efi partition that is needed by the arm64 vms).

Result

4 Likes

Woow! that seems impressive.

So you got the raspberry pi to be provisioned by MAAS?

I saw another thread which they had some success but the ROOT filesystem was mounted from NFS, which, from my point of view defeats part of the purpose of MAAS.

Are you flashing the root filesystem from MAAS? can you pass partitioning options, etc?

Some other people has comented on using a small PI and a rele board to power on/off, but MASS does not support power pluings yet.

It would be fantastic to have a proper MAAS RPI cluster.

1 Like

Thank you!

Yes, the raspberry pi is provisioned by MAAS like a normal computer. After installing the UEFI firmware on the sd card the pi becomes like a normal pc, you can enter the uefi bios to see all the connected drives, boot from them or from pxe, change the boot order or some other settings.

I have seen that thread too, but in that case, it uses the native network boot of the raspberry (not the uefi firmware one), which works differently, and I don’t think it can be used easily with maas.

I have not tried to partition the drives differently from the maas default (flat layout), but I think it should be possible given that the raspberry is seen as a normal pc. All the drives connected through usb are seen by maas and you can manage them from the UI (only the sd card is not visible because the uefi firmware is installed on it and I also believe that linux does not yet have the drivers to manage it).

The power management is a problem. For now, I had to set it to manual, but I don’t think there are easy alternatives since the raspberry doesn’t even have wake on lan. Maybe you could use ssh to shut it down or check the power status, but to turn it on I think you need something external like a relay.