Provisioning of Raspberry Pi 4 with MAAS

I recently played a bit with MAAS and I was trying to see how far I can go in provision my rpi4. I post these notes hoping they will turn out to be useful in the discussion about adding support for Raspberry Pi, or similar IoT boards, in MAAS.

FIrst, the reason for doing it is that I personally see a good fit for Metal As A Service for IoT devices. When it comes to the edge, the number of devices tends to increase as well the provisioning costs. Standards are not yet mature or not widely adopted, especially for booting technologies like PXE-boot, SBBR, etc. mainly caused by the fragmentation of the IoT ecosystem. All of this increase provisioning costs even more. This is why I believe MAAS could help to lower those costs in this area too.

Coming back to my experiment … the first thing to decide was: what I want to be provisioned. After some thinking, I decided to go first with Ubuntu classic (20.04 - 32 bits), then eventually move to Ubuntu Core depending on the results with classic.

Next, I installed MAAS 2.8 on my laptop. I decided to use my physical ethernet port for provisioning and I configured the network accordingly using the web interface. I configured DHCP on the subnet 10.10.50.0. I selected Ubuntu 20.04 LTS as image and armhf as architecture. I disabled automatically synch images.

Next, I’ve tried to understand how to network-boot my rpi4. I identified three ways to do it:

  1. with Uboot on SD card
  2. with UEFI on SD card
  3. with rpi First Stage Bootloader (EEPROM)

1. Network boot with uboot on SD card
I tried first this option since it seemed to be the easier one. Uboot is the bootloader of choice for Ubuntu on rpi, and uboot comes with built-in pxe support. The drawback is that an SD card is still required to network boot.
I prepared an SD card with Ubuntu and removed all unnecessary things (such the writable partition) just to be sure the system is not starting from SD, then I created a uboot.scr file as following:

fdt addr $fdt_addr
fdt move $fdt_addr $fdt_addr_r
setenv fdt_addr $fdt_addr_r
pxe get
pxe boot

Then I converted the file in uboot format:
mkimage -A arm -O linux -T script -C none -n uboot -d uboot.cfg uboot.scr

Unfortunately, uboot shipped with ubuntu does not support ethernet chip, so I had to compile uboot myself. I native compiled it on my rpi4. I took ubuntu on github:

git clone https://github.com/u-boot/u-boot.git
cd u-boot
make distclean
make rpi_4_defconfig
make all 

Once built and once copied on SD, I was able to network boot rpi, but with some limitations:

  • uboot ethernet driver seems not work properly at full speed. In order to boot I’d need to use an (old) 10/100 Mbps switch
  • the boot stopped after loading third state bootloader lpxelinux.0, which fails to load configuration file

2. Network boot with UEFI on SD card
UEFI seems to be a quite interesting solution: a SBBR-compliant UEFI porting on rpi is avaiable at rpi4-uefi.dev. SBBR (Server Base Boot Requirements) is a set of requirements aiming to reduce the adaptation efforts to boot an OS on a specific board. Ideally, an SBBR.compliant bootloader is able to boot an OS without the need of a custom kernel. The benefit, compared to uboot, is the standardisation behind and probably a better community support. Again, the drawback is that an SD card (with UEFI on it) is still required to network boot.

The result was promising: I’ve got no ethernet issues as with uboot, but it ended up the same way as uboot: the boot stopped after loading third state bootloader lpxelinux.0, which fails to load configuration file

3. Network boot with rpi First Stage Bootloader (EEPROM)
Lastly, I tried to take advantage of rpi first stage bootloader on EEPROM. FSB’s latest version has a (beta) PXE implementation, so the use of an SD card with a bootloader on it could be avoided. Unfortunately, this implementation is not fully compliant. In particular, it seems to ignore directions from MAAS in dhcp lease packets on where to get kernel, init.rd, etc.

I decided to do a step further. I had a look here, I tried to understand how a PXE server should be configured to support rpi4 network boot, and I tried to modify MAAS accordingly.
At the end, I was able to boot-up my rpi4 with focal (32-bit).
MAAS was able to provide firmware, kernel, kernel command line and initrd via PXE. For the root filesystem I decided to go with nfs, and I configured an nfs server for that purpose. So, the root filesystem part is still off control by MAAS, due to the lack of specific rpi images in MAAS.

Below what I’ve done:

3.1 Setting-up rpi4 for network boot
This procedure explains how to flash the latest EEPROM firmware and to configure the bootloader to start network boot when the SD card boot fails

  1. Insert a micro-sd card (>16GB) ready in the sd-card reader slot
  2. Install Raspberry Pi Imager

sudo snap install rpi-imager

  1. Choose OS -> Raspberry Pi OS (other) -> Raspberry Pi OS Lite (32-bit)
  2. Choose SD Card, then WRITE
  3. Boot your rpi, find the ip address (looking for lease in dhcp server log, in your router web page, etc.), then ssh with password: ‘raspberry’, then update and upgrade, then reboot
ssh pi@<ip_addr_of_your_rpi>
sudo apt-get update
sudo apt-get upgrade
sudo reboot
  1. Check the latest stable eeprom fw on github, project raspberrypi, rpi-eeprom/tree/master/firmware/stable (at the time of writing this doc, it was pieeprom-2020-07-16.bin and set the variable accordingly

PI_EEPROM_VERSION=pieeprom-2020-07-16

  1. Update eeprom with latest fw version, get the configuration, set the boot order and update again
wget https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/stable/${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-config ${PI_EEPROM_VERSION}.bin > bootconf.txt
sed -i 's/BOOT_ORDER=.*/BOOT_ORDER=0x21/g' bootconf.txt
sudo rpi-eeprom-config --out ${PI_EEPROM_VERSION}-netboot.bin --config bootconf.txt ${PI_EEPROM_VERSION}.bin
sudo rpi-eeprom-update -d -f ./${PI_EEPROM_VERSION}-netboot.bin
sudo reboot

NOTE! For some reasons, the wget command on Raspberry Pi OS results in an incomplete file (about 78KB against 524KB). In this case I had to download to my laptop first and then scp to rpi.

  1. Check version and boot parameters are as expected
$ vcgencmd bootloader_version
Jul 16 2020 16:15:46
version 45291ce619884192a6622bef8948fb5151c2b456 (release)
timestamp 1594912546
vcgencmd bootloader_config
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
ENABLE_SELF_UPDATE=1
DISABLE_HDMI=0
BOOT_ORDER=0x21

Note: BOOT_ORDER should be 0x21 to allow network boot.

3.2 Compilation and installation of MAAS on Ubuntu 20.04

  1. Create a new wired connection, static ip 10.10.50.1 netmask 255.255.255.0, gateway 10.10.50.1
  2. Connect RPI eth to laptop eth with an ethernet cable (or through a switch)
  3. Get MAAS sources
git clone --recurse-submodules https://git.launchpad.net/maas maas-rpi
cd maas-rpi
  1. Patch MAAS
    We need to modify the MAAS dhcp server configuration in order to add in the dhcp offer lease specific options (PXE.discovery-control, PXE.boot-menu and PXE.menu-prompt) which triggers rpi4 FSB to start tftp download of firmware, kernel and initrd images.

Create the patch file: copy in the file the following text and save as maas-rpi.patch

diff --git a/src/provisioningserver/dhcp/config.py b/src/provisioningserver/dhcp/config.py
index 717c6e4..f3f8c4d 100644
--- a/src/provisioningserver/dhcp/config.py
+++ b/src/provisioningserver/dhcp/config.py
@@ -27,7 +27,7 @@
 
 logger = logging.getLogger(__name__)
 
-
+    
 # Used to generate the conditional bootloader behaviour
 CONDITIONAL_BOOTLOADER = tempita.Template(
     """
@@ -102,6 +102,11 @@
     {{if http_client}}
     option vendor-class-identifier "HTTPClient";
     {{endif}}
+    option vendor-class-identifier \"PXEClient\";
+        vendor-option-space PXE;
+            option PXE.discovery-control 3;
+            option PXE.boot-menu 0 17 \"Raspberry Pi Boot\";
+            option PXE.menu-prompt 0 \"PXE\";
 }
 {{endif}}
 {{endif}}
@@ -136,6 +141,11 @@
             option dhcp-parameter-request-list,d2);
     }
     {{endif}}
+    option vendor-class-identifier \"PXEClient\";
+        vendor-option-space PXE;
+            option PXE.discovery-control 3;
+            option PXE.menu-prompt 0 \"PXE\";
+            option PXE.boot-menu 0 17 \"Raspberry Pi Boot\";
 }
 {{endif}}
 """

To apply the patch:
patch -p1 < maas-rpi.patch

  1. Make MAAS snap
make install-dependencies
make snap

Note: in case you are building on focal, make install-dependencies may fail: this is not relevant since we are building the snap via lxd

  1. Install maas, connect interfaces and install the postgres db
sudo snap install maas_2.8.0~alpha1-8271-g.0acd342d4_amd64.snap --dangerous
./utilities/connect-snap-interfaces
sudo snap restart maas
sudo snap install maas-test-db
  1. Test the db (not sure this is really needed)
sudo maas-test-db.psql
psql (10.6)
Type "help" for help.
postgres=#
  1. Initialize MAAS
sudo maas init
Mode (all/region+rack/region/rack/none) [default=all]?
MAAS URL [default=http://10.10.50.1:5240/MAAS]:
Create first admin account
Username: admin
Password:
Again:
Email: admin@example.com
Import SSH keys [] (lp:user-id or gh:user-id): lp:yourusername

In case the SSH keys import fails, don’t worry and go on: it can be retrieved lately with the MAAS dashboard.

  1. Log in to the maas UI (port 5240) with the user just created, and complete the first time setup, ensuring to select bionic “armhf” images (“amd64” ones can be deselected)

  2. Under the “Images” tab, deselect “Automatically sync images” (top-right corner)

  3. Under the “Subnets” tab, click on the VLAN associated with the subnet you want MAAS to provide DHCP for (10.10.50.0 in our example), and select “Enable DHCP” in the page, confirming the DHCP range to use.

  4. Configure firewall

sudo ufw allow 53/tcp
sudo ufw allow 53/udp
sudo ufw allow 67/udp
sudo ufw allow 69/udp
sudo ufw allow 4011/udp

3.3 MAAS configuration extra-steps
In this section I describe how to add to MAAS the rpi boot images and modify configuration files. First, we need a SD card with focal

  1. Insert a micro-sd card (>16GB) ready in the sd-card reader slot
  2. Install Raspberry Pi Imager

sudo snap install rpi-imager

  1. Choose OS -> Ubuntu -> UBuntu 20.04 LTS (32-bit)
  2. Choose SD Card, then WRITE
  3. Insert SD Card in card reader on your pc
  4. Copy boot images on MAAS boot.-resources
sudo mkdir /var/snap/maas/current/var/lib/maas/boot-resources/current/be53147
sudo cp <path-to-system-boot>/* /var/snap/maas/current/var/lib/maas/boot-resources/current/be53147
  1. Edit config.txt. Replace [pi4] section with this:
[pi4]
kernel=vmlinuz
max_framebuffers=2
initramfs initrd.img followkernel

3.4 NFS server configuration
To setup a NFS server follow these steps.
Additionally remember to disable your firewall (sudo ufw disable), or even better, limit nfsd and mountd ports usage, and reconfigure ufw accordingly
As a last step, be sure that your kernel commandline looks like this:

cd /var/snap/maas/current/var/lib/maas/boot-resources/current/be531479
cat cmdline.txt
initrd=-1 net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline root=/dev/nfs nfsroot=10.10.50.1:/nfs-export/PI-Ubuntu,tcp,rw rootfstype=nfs rootwait fixrtc 

At the end of the steps above, connect your rpi4 with your pc with an ethernet cable, be sure no SD card is inserted, switch it on: you should see the EEPROM bootloader getting an ip from MAAS dhcp server, then the bootloader loading start4.elf, config.txt, vmlinuz, cmdline.txt, initrd.img and boot the kernel. After getting again an IP, the root filesystem should be mounted and after a while the login prompt should pop up.

6 Likes

@dbruno74, very interesting post; i will have to study it closely in a little while, looks very promising at first glance.

@dbruno74 - I suspect there may be an opportunity to simplify some of this, if not solve the outcome of (at least) #2 (UEFI)

You mention that with UEFI, and even with #1, you wound up landing on “Unable to locate configuration file.”

We actually ran into that, too. It was due to using lpxelinux0.cfg with UEFI - for which you’d need pxelinux0.cfg. Then we ran into it with the inverse: our nodes were reverted back to non-UEFI boot mode (“Legacy - non-UEFI BIOS”) while PXE traffic was pinned to pxelinux0.cfg. Both resulted in the “Cannot load configuration file” error.

Once everything lined up (UEFI BIOS + pxelinux0.cfg + appropriate arch,# tagged for PXE traffic), everything worked.

Furthermore, we found it much simpler to work with MAAS when we own the DHCP/DNS process (instead of using MAAS-DHCP/DNS) as MAAS would compete with our fairly hands-off DHCP implementation.

I wonder: if you defined the appropriate arch type (perhaps arch,11) in your dnsmasq.conf (or however you’re forwarding PXE traffic to MAAS) and tagged that to make use of pxelinux0.cfg (when UEFI-booting the rpi4), would you find success where previously there were hurdles?

Follow-up Question

@dbruno74 - In terms of PXE-booting/MAAS-deploying an rpi4, what would you recommend doing if we actually do want / prefer an SD card in the rpi4?

We’d essentially prefer to treat the rpi4 as if it’s a “real” machine - CPU/GPU/RAM/storage, all on-board. A tiny PC. A mini-node.

It sounds like #1 is what would fit for us - but what do you think? I just want to PXE boot our rpi4 kits with MAAS, Enlist / Commission / Deploy them - and we don’t mind if “Power Control” has to be “manual”.

@knaledge, thanks for your answer, pretty interesting.

Totally makes sense: using an external DHCP/DNS server could eventually solve the PXE problem, especially when dealing with non-standard booting process (such rpi has). But even with an external DHCP/DNS server I’m afraid MAAS wouldn’t be (yet) ready to commission rpi because of the lack of an enlisting/commissioning image able to run on the rpi.

Regarding your questions: using or not an SD card it really depends on your use case. If we think about a deployment in the field (and this was mainly the use case I had in mind), an SD card can be removed, or stolen, or can move accidentally (e.g. due to vibrations in the environment where the device is installed), so in this case an ephemeral image running on ram and a distributed storage like Ceph for persistency could be best suited. In case you need to prime the device with an OS, and have persistent storage onboard, an SD card would be required.

Regarding option #1, I believe it is more tricky, since you need to recompile u-boot, and for some reason, the eth driver doesn’t work properly. In case you go with SD, I’ll would recommend UEFI instead … let us know how it goes!

@knaledge, would you consider writing up a guide if you do get a UEFI network boot Pi cluster going? A MAAS-managed Pi cluster is my dream but I’m not too confident in my networking knowledge to pull it off.

Very interested in your tests.
I have not yet started on the MAAS side but have already managed to boot Ubuntu 20.04 LTS (64Bits) using UEFI. One of the limitations at this stage is that Ubuntu 20.04 Kernel version does not contain the necessary Network drives, so while Ubuntu will boot it will not communicate, at least on the onboard network adapters. Debian 11 is the way to go if you want the network adapter supported but did not manage the boot for other reasons.
I would very much like to find a solution to boot from network without a SD Card is that there is no way to secure the boot, there is no ipxe support and thus the image pushed by the server can not be secured. As you are considering use of MAAS for Edge devices, have you looked at the security aspect of booting the RSPI on a potentially unsafe network ?
Very interested in your thoughts on the matter.
Thanks for Sharing
Raymond