I recently played a bit with MAAS and I was trying to see how far I can go in provision my rpi4. I post these notes hoping they will turn out to be useful in the discussion about adding support for Raspberry Pi, or similar IoT boards, in MAAS.
FIrst, the reason for doing it is that I personally see a good fit for Metal As A Service for IoT devices. When it comes to the edge, the number of devices tends to increase as well the provisioning costs. Standards are not yet mature or not widely adopted, especially for booting technologies like PXE-boot, SBBR, etc. mainly caused by the fragmentation of the IoT ecosystem. All of this increase provisioning costs even more. This is why I believe MAAS could help to lower those costs in this area too.
Coming back to my experiment ā¦ the first thing to decide was: what I want to be provisioned. After some thinking, I decided to go first with Ubuntu classic (20.04 - 32 bits), then eventually move to Ubuntu Core depending on the results with classic.
Next, I installed MAAS 2.8 on my laptop. I decided to use my physical ethernet port for provisioning and I configured the network accordingly using the web interface. I configured DHCP on the subnet 10.10.50.0. I selected Ubuntu 20.04 LTS as image and armhf as architecture. I disabled automatically synch images.
Next, Iāve tried to understand how to network-boot my rpi4. I identified three ways to do it:
- with Uboot on SD card
- with UEFI on SD card
- with rpi First Stage Bootloader (EEPROM)
1. Network boot with uboot on SD card
I tried first this option since it seemed to be the easier one. Uboot is the bootloader of choice for Ubuntu on rpi, and uboot comes with built-in pxe support. The drawback is that an SD card is still required to network boot.
I prepared an SD card with Ubuntu and removed all unnecessary things (such the writable partition) just to be sure the system is not starting from SD, then I created a uboot.scr file as following:
fdt addr $fdt_addr
fdt move $fdt_addr $fdt_addr_r
setenv fdt_addr $fdt_addr_r
pxe get
pxe boot
Then I converted the file in uboot format:
mkimage -A arm -O linux -T script -C none -n uboot -d uboot.cfg uboot.scr
Unfortunately, uboot shipped with ubuntu does not support ethernet chip, so I had to compile uboot myself. I native compiled it on my rpi4. I took ubuntu on github:
git clone https://github.com/u-boot/u-boot.git
cd u-boot
make distclean
make rpi_4_defconfig
make all
Once built and once copied on SD, I was able to network boot rpi, but with some limitations:
- uboot ethernet driver seems not work properly at full speed. In order to boot Iād need to use an (old) 10/100 Mbps switch
- the boot stopped after loading third state bootloader lpxelinux.0, which fails to load configuration file
2. Network boot with UEFI on SD card
UEFI seems to be a quite interesting solution: a SBBR-compliant UEFI porting on rpi is avaiable at rpi4-uefi.dev. SBBR (Server Base Boot Requirements) is a set of requirements aiming to reduce the adaptation efforts to boot an OS on a specific board. Ideally, an SBBR.compliant bootloader is able to boot an OS without the need of a custom kernel. The benefit, compared to uboot, is the standardisation behind and probably a better community support. Again, the drawback is that an SD card (with UEFI on it) is still required to network boot.
The result was promising: Iāve got no ethernet issues as with uboot, but it ended up the same way as uboot: the boot stopped after loading third state bootloader lpxelinux.0, which fails to load configuration file
3. Network boot with rpi First Stage Bootloader (EEPROM)
Lastly, I tried to take advantage of rpi first stage bootloader on EEPROM. FSBās latest version has a (beta) PXE implementation, so the use of an SD card with a bootloader on it could be avoided. Unfortunately, this implementation is not fully compliant. In particular, it seems to ignore directions from MAAS in dhcp lease packets on where to get kernel, init.rd, etc.
I decided to do a step further. I had a look here, I tried to understand how a PXE server should be configured to support rpi4 network boot, and I tried to modify MAAS accordingly.
At the end, I was able to boot-up my rpi4 with focal (32-bit).
MAAS was able to provide firmware, kernel, kernel command line and initrd via PXE. For the root filesystem I decided to go with nfs, and I configured an nfs server for that purpose. So, the root filesystem part is still off control by MAAS, due to the lack of specific rpi images in MAAS.
Below what Iāve done:
3.1 Setting-up rpi4 for network boot
This procedure explains how to flash the latest EEPROM firmware and to configure the bootloader to start network boot when the SD card boot fails
- Insert a micro-sd card (>16GB) ready in the sd-card reader slot
- Install Raspberry Pi Imager
sudo snap install rpi-imager
- Choose OS ā Raspberry Pi OS (other) ā Raspberry Pi OS Lite (32-bit)
- Choose SD Card, then WRITE
- Boot your rpi, find the ip address (looking for lease in dhcp server log, in your router web page, etc.), then ssh with password: āraspberryā, then update and upgrade, then reboot
ssh pi@<ip_addr_of_your_rpi> sudo apt-get update sudo apt-get upgrade sudo reboot
- Check the latest stable eeprom fw on github, project raspberrypi, rpi-eeprom/tree/master/firmware/stable (at the time of writing this doc, it was pieeprom-2020-07-16.bin and set the variable accordingly
PI_EEPROM_VERSION=pieeprom-2020-07-16
- Update eeprom with latest fw version, get the configuration, set the boot order and update again
wget https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/stable/${PI_EEPROM_VERSION}.bin sudo rpi-eeprom-config ${PI_EEPROM_VERSION}.bin > bootconf.txt sed -i 's/BOOT_ORDER=.*/BOOT_ORDER=0x21/g' bootconf.txt sudo rpi-eeprom-config --out ${PI_EEPROM_VERSION}-netboot.bin --config bootconf.txt ${PI_EEPROM_VERSION}.bin sudo rpi-eeprom-update -d -f ./${PI_EEPROM_VERSION}-netboot.bin sudo reboot
NOTE! For some reasons, the wget command on Raspberry Pi OS results in an incomplete file (about 78KB against 524KB). In this case I had to download to my laptop first and then scp to rpi.
- Check version and boot parameters are as expected
$ vcgencmd bootloader_version Jul 16 2020 16:15:46 version 45291ce619884192a6622bef8948fb5151c2b456 (release) timestamp 1594912546 vcgencmd bootloader_config [all] BOOT_UART=0 WAKE_ON_GPIO=1 POWER_OFF_ON_HALT=0 DHCP_TIMEOUT=45000 DHCP_REQ_TIMEOUT=4000 TFTP_FILE_TIMEOUT=30000 ENABLE_SELF_UPDATE=1 DISABLE_HDMI=0 BOOT_ORDER=0x21
Note: BOOT_ORDER should be 0x21 to allow network boot.
3.2 Compilation and installation of MAAS on Ubuntu 20.04
- Create a new wired connection, static ip 10.10.50.1 netmask 255.255.255.0, gateway 10.10.50.1
- Connect RPI eth to laptop eth with an ethernet cable (or through a switch)
- Get MAAS sources
git clone --recurse-submodules https://git.launchpad.net/maas maas-rpi cd maas-rpi
- Patch MAAS
We need to modify the MAAS dhcp server configuration in order to add in the dhcp offer lease specific options (PXE.discovery-control, PXE.boot-menu and PXE.menu-prompt) which triggers rpi4 FSB to start tftp download of firmware, kernel and initrd images.
Create the patch file: copy in the file the following text and save as maas-rpi.patch
diff --git a/src/provisioningserver/dhcp/config.py b/src/provisioningserver/dhcp/config.py index 717c6e4..f3f8c4d 100644 --- a/src/provisioningserver/dhcp/config.py +++ b/src/provisioningserver/dhcp/config.py @@ -27,7 +27,7 @@ logger = logging.getLogger(__name__) - + # Used to generate the conditional bootloader behaviour CONDITIONAL_BOOTLOADER = tempita.Template( """ @@ -102,6 +102,11 @@ {{if http_client}} option vendor-class-identifier "HTTPClient"; {{endif}} + option vendor-class-identifier \"PXEClient\"; + vendor-option-space PXE; + option PXE.discovery-control 3; + option PXE.boot-menu 0 17 \"Raspberry Pi Boot\"; + option PXE.menu-prompt 0 \"PXE\"; } {{endif}} {{endif}} @@ -136,6 +141,11 @@ option dhcp-parameter-request-list,d2); } {{endif}} + option vendor-class-identifier \"PXEClient\"; + vendor-option-space PXE; + option PXE.discovery-control 3; + option PXE.menu-prompt 0 \"PXE\"; + option PXE.boot-menu 0 17 \"Raspberry Pi Boot\"; } {{endif}} """
To apply the patch:
patch -p1 < maas-rpi.patch
- Make MAAS snap
make install-dependencies make snap
Note: in case you are building on focal, make install-dependencies may fail: this is not relevant since we are building the snap via lxd
- Install maas, connect interfaces and install the postgres db
sudo snap install maas_2.8.0~alpha1-8271-g.0acd342d4_amd64.snap --dangerous ./utilities/connect-snap-interfaces sudo snap restart maas sudo snap install maas-test-db
- Test the db (not sure this is really needed)
sudo maas-test-db.psql psql (10.6) Type "help" for help. postgres=#
- Initialize MAAS
sudo maas init Mode (all/region+rack/region/rack/none) [default=all]? MAAS URL [default=http://10.10.50.1:5240/MAAS]: Create first admin account Username: admin Password: Again: Email: admin@example.com Import SSH keys [] (lp:user-id or gh:user-id): lp:yourusername
In case the SSH keys import fails, donāt worry and go on: it can be retrieved lately with the MAAS dashboard.
-
Log in to the maas UI (port 5240) with the user just created, and complete the first time setup, ensuring to select bionic āarmhfā images (āamd64ā ones can be deselected)
-
Under the āImagesā tab, deselect āAutomatically sync imagesā (top-right corner)
-
Under the āSubnetsā tab, click on the VLAN associated with the subnet you want MAAS to provide DHCP for (10.10.50.0 in our example), and select āEnable DHCPā in the page, confirming the DHCP range to use.
-
Configure firewall
sudo ufw allow 53/tcp sudo ufw allow 53/udp sudo ufw allow 67/udp sudo ufw allow 69/udp sudo ufw allow 4011/udp
3.3 MAAS configuration extra-steps
In this section I describe how to add to MAAS the rpi boot images and modify configuration files. First, we need a SD card with focal
- Insert a micro-sd card (>16GB) ready in the sd-card reader slot
- Install Raspberry Pi Imager
sudo snap install rpi-imager
- Choose OS ā Ubuntu ā UBuntu 20.04 LTS (32-bit)
- Choose SD Card, then WRITE
- Insert SD Card in card reader on your pc
- Copy boot images on MAAS boot.-resources
sudo mkdir /var/snap/maas/current/var/lib/maas/boot-resources/current/be53147 sudo cp <path-to-system-boot>/* /var/snap/maas/current/var/lib/maas/boot-resources/current/be53147
- Edit config.txt. Replace [pi4] section with this:
[pi4] kernel=vmlinuz max_framebuffers=2 initramfs initrd.img followkernel
3.4 NFS server configuration
To setup a NFS server follow these steps.
Additionally remember to disable your firewall (sudo ufw disable), or even better, limit nfsd and mountd ports usage, and reconfigure ufw accordingly
As a last step, be sure that your kernel commandline looks like this:
cd /var/snap/maas/current/var/lib/maas/boot-resources/current/be531479 cat cmdline.txt initrd=-1 net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline root=/dev/nfs nfsroot=10.10.50.1:/nfs-export/PI-Ubuntu,tcp,rw rootfstype=nfs rootwait fixrtc
At the end of the steps above, connect your rpi4 with your pc with an ethernet cable, be sure no SD card is inserted, switch it on: you should see the EEPROM bootloader getting an ip from MAAS dhcp server, then the bootloader loading start4.elf, config.txt, vmlinuz, cmdline.txt, initrd.img and boot the kernel. After getting again an IP, the root filesystem should be mounted and after a while the login prompt should pop up.