Raspberry Pi 4 provisioning and KVM pod setup

Hi, these days I have been trying to provision a raspberry pi 4 through MAAS and I did it. I did that using the Raspberry Pi 4 UEFI Firmware, but I had to fix a few problems.

Hardware

I used a raspberry pi 4 8 gb, a micro sd card for the UEFI firmware and an usb drive (where the os will be installed).

UEFI Firmware Setup

The Raspberry Pi 4 UEFI Firmware can perform a pxe boot, but currently (version 1.21) there are some problems that need to be fixed to work properly with MAAS:

  1. Versions after 1.16 do not work. Apparently, a commit related to version 1.17 broke the network: the uefi firmware can download the grub bootloader, but then It always shows the error message “couldn’t send network packet”. In grub most of the commands related to network (like net_dhcp) show that error. However, version 1.16 works, and it managed to download kernel files. I opened an issue.
  2. After the installation of ubuntu, maas sends to the client a different configuration file, that tells to scan every device to find the bootloader. However, when the rpi4 uefi firmware boots normally, it does not attach every device, but only the one from which was set to boot from. So, the first reboot after the deploy step fails, because the os bootloader cannot be found. It is possible to avoid this by pressing esc to enter the uefi setup and then press continue. By doing this all device are connected and the os can be booted, but you need to do it manually every time. I opened an issue.

I solved these problems recompiling the firmware, using the network drivers from version 1.16 and adding the line removed from that commit.

As for the uefi firmware configuration, I removed the 3 gb limit (Device Manager -> Raspberry Pi Configuration -> Advanced Configuration -> Limit RAM to 3 GB -> Disabled) and set the boot order to start from the network (Boot Maintenance Manager -> Boot Options -> Change Boot Order -> UEFI PXEv4 on top).

MAAS Configuration

I used snap MAAS 2.9/candidate.

In the MAAS UI, I selected to download ubuntu 20.04 and 20.10 for arm64 and I setup dhcp normally. I also set in the settings ubuntu 20.04 with the ga-20.04 minimum kernel as the default ubuntu release for commissioning and ubuntu 20.10 for deployment.

To work properly, the rpi4 with the UEFI firmware needs the 5.8 linux kernel (otherwise the network will not work, and the RAM must be limited to 3 GB). That kernel is present only in ubuntu groovy (20.10). MAAS can deploy ubuntu groovy, but the listing and commissioning steps can use, at most, ubuntu focal (that has linux kernel 5.4). The fastest solution that I found to solve this problem is to replace the ubuntu focal image files with the groovy ones. I used the mount and unmount commands to mount the groovy folder on top of the focal one, to quickly change images by mounting and unmounting folders.

I disabled the image autosync in MAAS and then replaced the image files:

sudo mount --bind -o nodev,ro /var/snap/maas/common/maas/boot-resources/snapshot–/ubuntu/arm64/ga-20.10/groovy/stable /var/snap/maas/common/maas/boot-resources/snapshot-/ubuntu/arm64/ga-20.04/focal/stable

After that, the raspberry can be powered on and the listing and commissioning step should succeed (using ubuntu 20.04 that in reality is ubuntu 20.10).
After the commissioning, the images can be reverted by unmounting the folder:

sudo umount /var/snap/maas/common/maas/boot-resources/snapshot-/ubuntu/arm64/ga-20.04/focal/stable

I do not know if there is a better way to commission a machine with linux kernel 5.8. A good solution would be to update the kernel of the 20.04 image, so that it would also possible to deploy ubuntu 20.04 directly on the raspberry, that is better supported (also by juju bundles that are all for focal and not for groovy). I do not know if that could be possible.

KVM Pod setup

After the commissioning, ubuntu 20.10 can be deployed.

Currently from the UI is not possible to deploy a node with ubuntu 20.10 as a kvm pod. That must be done from the cli:

maas $PROFILE machine deploy $SYSTEM_ID install_kvm=True

Since the default ubuntu release for the deployment was set to ubuntu 20.10, that release will be deployed.

After the deployment, the kvm pod will show up in the UI, but when trying to compose a machine il will throw an error. That is caused by the virsh template used by MAAS to compose the virtual machine for arm64, that apparently is not suitable for the rpi4.

The file /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod/virsh.py must be changed. However, that file is inside snap and cannot be changed. I solved this problem by mounting another folder with the same files on top of it.

I copied the folder with these files in my home directory:

sudo cp -r /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod ~/pod

Changed owner to my user to edit them:

sudo chown -R <you_user> ~/pod

Edited the file ~/pod/virsh.py. The template “DOM_TEMPLATE_ARM64” must be changed as follow:

  • Change <gic version='3'/> to <gic version='2'/>
  • Change <input type='mouse' bus='ps2'/> to <input type='mouse' bus='virtio'/>
  • Change <input type='keyboard' bus='ps2'/> to <input type='keyboard' bus='virtio'/>

Mount the folder on top of the snap one:

sudo mount --bind -o nodev,ro ~/pod /snap/maas/current/lib/python3.8/site-packages/provisioningserver/drivers/pod

Restart MAAS:

sudo snap restart maas.supervisor

The folder must be remounted every time the MAAS controller is rebooted if you want to compose a new vm.

After these steps it should be possible to compose some virtual machines on the rpi4, using any release of ubuntu arm64.

Deploy JUJU bundles

I tried to deploy the Kubernetes core bundle and it works. However, you must first create the virtual machines on maas and then deploy the juju bundle. I don’t know why, but if you don’t do that, juju will create the vm, but it will misconfigure the storage (not creating the fat32 efi partition that is needed by the arm64 vms).

Result

5 Likes

Woow! that seems impressive.

So you got the raspberry pi to be provisioned by MAAS?

I saw another thread which they had some success but the ROOT filesystem was mounted from NFS, which, from my point of view defeats part of the purpose of MAAS.

Are you flashing the root filesystem from MAAS? can you pass partitioning options, etc?

Some other people has comented on using a small PI and a rele board to power on/off, but MASS does not support power pluings yet.

It would be fantastic to have a proper MAAS RPI cluster.

1 Like

Thank you!

Yes, the raspberry pi is provisioned by MAAS like a normal computer. After installing the UEFI firmware on the sd card the pi becomes like a normal pc, you can enter the uefi bios to see all the connected drives, boot from them or from pxe, change the boot order or some other settings.

I have seen that thread too, but in that case, it uses the native network boot of the raspberry (not the uefi firmware one), which works differently, and I don’t think it can be used easily with maas.

I have not tried to partition the drives differently from the maas default (flat layout), but I think it should be possible given that the raspberry is seen as a normal pc. All the drives connected through usb are seen by maas and you can manage them from the UI (only the sd card is not visible because the uefi firmware is installed on it and I also believe that linux does not yet have the drivers to manage it).

The power management is a problem. For now, I had to set it to manual, but I don’t think there are easy alternatives since the raspberry doesn’t even have wake on lan. Maybe you could use ssh to shut it down or check the power status, but to turn it on I think you need something external like a relay.

It would be good if MAAS adds support for the groovy version. All the other bits and pieces seem to be coming along.

I am thinking on creating an specific connected power supply for RPIs, ammong others.

Based on the ESP32, for which there is Ethernet support and many openhardware projects + some reles.

I also want the same setup to power other devices such as the odroid-h2, that I am using for creating maas clusters.

I saw yesterday that they added the hwe-20.04-edge kernel to ubuntu 20.04 images. That version has the linux kernel 5.8, so using that it is possible to commission without changing the images. Then it is possible to deploy ubuntu 20.04 or 20.10 directly on the raspberry.

I was thinking of building a BMC through an ESP32 too. That is the next thing I will try to do. I have also seen that the raspberry can be woken up through the gpio (shorting GPIO3 to ground). Maybe that could be useful to avoid using relays.

1 Like

interesting.

I am putting all of this together in a domestic blade center. Still working on it, you can see some details here:

https://magazine.odroid.com/article/building-an-odroid-h2-bladecenter-create-a-micro-footprint-high-performance-computing-station/

Thou this version is for the h2.

lets stay in touch and see if we can cooperate.

1 Like

Sure! I will keep you updated on my progress.

1 Like

Have you made any more progress on this? I’m just getting started with maas and messed around with some vms in proxmoxve and realized how important the power driver would be. As a result I took the time to write a webhook driver which sends commands to a given endpoint where you can then do as you wish. In my use-case the intent has been to push to commands to Node-RED which in turn sends commands to cheap tasmota based power plugs via home assistant. It’s been prototyped and works wonderfully. It would be easy to interface with web-connected relays or whatever if tasmota/home assistant doesn’t meet your needs.

Now with that out of the way I’m moving to the next step of trying to actually manage an rpi. In my situation I want to control the device like any other server and install right to the sdcard (ie: not nfs) so any pointers you can provide would be great!

A few weeks ago, I tried to do something similar to what you did. I had made a simple implementation of a power driver for maas which calls some kind of API on an esp32 (but it could be anything else). Then the esp32 can read the state of a pin or switch a relay. It seemed to work but it was just a test implementation.

As for the raspberry, if you use the uefi firmware you cannot install the operating system on the sd card but only on an external disk. I don’t know if it’s possible to avoid using uefi, I have my doubts.

This is amazing, I hope it works for me. If I can automate my K8s cluster then I’m going to send you some money:) or a coffee maybe:)

Awesome instructions! I ran into all the same problems (and some more) in my recent attempt in even July 2021!

Just want to share a few notes on worked for me in July 2021:

  1. We can now use hwe-20.04 (Kernel 5.8) instead of the groovy image trick. This way we can deploy KVM using the web UI as well.
  2. MaaS is not recognizing USB flash drives for UEFI firmware v1.16. So I’m using v1.28. PXE still doesn’t work, but we can use the HTTP v4 boot option (move this to the top of boot order rather than PXE v4). Apparently HTTP network boot also works with MaaS.
  3. Have to pick USB flash drive as boot drive (rather than the default SD card) in MaaS web UI to deploy to the correct drive. After KVM deployment, MaaS somehow wipes my micro SD, so I re-copied the UEFI v1.28 and re-setup UEFI with (>3GB memory, and change boot order). Then, it works!

My setup:

  • MaaS v3.0.0
  • RPi UEFI v1.28
  • Ubuntu HWE-20.04 (Kernel 5.8)

hi @someo_new (and anyone else interested in this) - I’m the PM for MAAS, and I’m glad that you managed to solve the MAAS side issues you had there.

I’m wondering if you would be interested to author an article about what you did (tutorial)? If you are, then we could link to it from our new resources page that we are building for MAAS.

I realise that it’s some work, but I’d love to have it out there for everyone and to make it more visible.

We also have a standard way to write tutorials for ubuntu.com if you wanted to host it there.

Let me know and I can loop in our RPI and Desktop product managers too :slight_smile: