we’re spinning up a bunch of small DCs around the world with a handful of machines, using maas to provision. right now, we configure a single drive with a single set of partitions, and then have a shell script that converts it to a mirrored setup I’ll document below. it seems like this new custom storage feature will get us a lot closer to what we want, and require a lot less scripting at the backend (although it still requires some because of the limited support for filesystem and zfs creation options).
our end target for our servers, which have 2x256GB NVMe drives, is both drives partitioned as the following:
part1: 128 MiB, type EF00
part2: 1GiB, type 8300
part3: 4GiB, type 8300
part4: rest of the drive, type 8300
then on top of that, we have:
efi: mdadm raid1 with v1.0 metadata, part1 on both drives, formatted vfat, mounted at /boot/efi
boot: mdadm raid1, part2 on both drives, formatted xfs, mounted at /boot
swap: mdadm raid1, part3 on both drives, formatted as encrypted swap using cryptsetup
rpool: encrypted zfs mirrored root pool, part4 on both drives
prior to the custom storage support, we had to use a custom curtin script we injected into the server, which had a lot of downsides, including not being portable across device names, and not having accurate storage configuration represented in the maas interface. with the custom storage support, we should be able to solve both of those things, install to a root xfs on the first drive on partition 4, and then migrate to ZFS, which significantly cuts down on the amount of scripting I have to do.
ideally, there’d be support for arbitrary zfs pools, and arbitrary filesystem and raid creation options, so we can do the efi metadata and the zfs root pool all without intervention. but if I can get this custom storage scripting working, we’ll at least be further along and the state in maas will be a lot more accurate for what’s actually on the systems.
it’s worth noting that the lack of any encrypted filesystem support in maas is a pretty huge deficiency, since it’s a hard requirement for anyone running in shared datacenters, thanks to things like the GDPR. basically, we can’t build from bare metal using MAAS and be GDPR compliant right now, so this is something I’d definitely like to see the team address.
as well, network config is still an annoying manual chore, and I’m highly looking forward to the day when we can have network templates that we can apply to arbitrary interfaces, so I don’t have to manually configure all our network settings on every host. basically we have vlans layered on bonding layered on physical nics (which is a pretty common setup), and doing all of that by hand for each host is a real chore.