Deploying servers with full disk encryption (LUKS2)

Hi everybody,

I recently had to deploy a few “full disk encrypted” servers with MAAS. While this is not something that’s supported officially, it’s relatively easy to do with preseeds. And once it’s done, you can scale out pretty easily.

An overview of the steps to follow are :

  1. get the server in MAAS and configure it
  2. fetch the curtin config
  3. create the preseed
  4. deploy

Requirements

You will require functional MAAS CLI. See https://maas.io/docs/maas-cli for more info.

Enlist, commission, configure

First, enlist and commission your server, and then configure its storage and networking as normal.

Fetch the curtin config

We’re now going to grab the curtin config that MAAS generates for our server. Since this depends on the OS you deploy, MAAS won’t give you said config unless the server is in a “deployment” state. So start a deploy of the target OS,
and then run the following from the MAAS CLI :

$ maas <profile> machine get-curtin-config <machine-id> > /tmp/curtin-config

You can now cancel the deploy.

IMPORTANT : if you already have a preseed matching your server deployment, the command above will return your preseed and not the MAAS-generated curtin config. Make sure that you do not have a preseed matching your server deployment when running the command above.

Create the preseed

Preseeds offer you a way to override parts of the default installation process. In this case, we’re going to rework the storage setup part.

Note that a preseed completely overwrites the MAAS-generated curtin config, so it needs to be “complete”. If you only put a storage: block in your preseed, the MAAS deploy will fail.

Open /tmp/curtin-config in an editor. Keep only the block starting by storage:. You’re going to recognize the storage you configured in MAAS. This YAML list can be a bit frightening at the beginning, but if you take a close look you’ll notice
that the list is sorted on the type key.

  • First, type: disk. This sets up disks for use by curtin.
  • Then, type: partition. This creates partitions on the disks.
  • Then, type: format. This formats the partitions with the specified filesystem.
  • And finally, type: mount. This mounts the filesystem on a mountpoint.

For the sake of this example, I’m going to assume that each LUKS device will only have one filesystem on it. curtin can create LUKS volumes using the dm_crypt type. We’re going to insert this between the partition (or disk if you want
to encrypt a full disk) and format stanzas.

For each partition block, you’re going to add a dm_crypt stanza, and then modify the corresponding format stanza to reference the encrypted partition.

Let’s take an example. Assume you have a /dev/sda3 partition that you want to encrypt. MAAS would likely have generated something like that :

storage:
  config:
  [... some disk/partitions stanzas ...]
  - device: sda
    id: sda-part3
    name: sda-part3
    number: 3
    size: 599085023232B
    type: partition
    uuid: 10a1b6a4-f31c-4502-abb0-537ba788fd2a
    wipe: superblock
  [... some more stuff possibly ...]
  - fstype: ext4
    id: sda-part3_format
    label: ''
    type: format
    uuid: 2ae7cb24-fcd4-4985-8f7b-7ed1752ccd2f
    volume: sda-part3
  [... even more stuff maybe ...]

Right. So as per above, we’re going to add a dm_crypt entry between these two, and modify the format stanza to refer to the dm_crypt entry. This would give ;

storage:
  config:
  [... some disk/partitions stanzas ...]
  - device: sda
    id: sda-part3
    name: sda-part3
    number: 3
    size: 599085023232B
    type: partition
    uuid: 10a1b6a4-f31c-4502-abb0-537ba788fd2a
    wipe: superblock
  [... some more stuff possibly ...]
  - id: sda-part3_crypt
    type: dm_crypt
    dm_name: sda3_crypt
    volume: sda-part3
    key: tempkey
    keysize: '512'
  - fstype: ext4
    id: sda-part3_format
    label: ''
    type: format
    uuid: 2ae7cb24-fcd4-4985-8f7b-7ed1752ccd2f
    volume: sda-part3_crypt
  [... even more stuff maybe ...]

I didn’t touch the partition block. I added the dm_crypt block, which is referring to the partition item via volume: sda-part3. And then I updated the format block, just replacing volume: sda-part3 by volume: sda-part3_crypt.

If you want to encrypt a full disk, it’s pretty much the same.
MAAS-generated curtin config would be :

storage:
  config:
  - id: nvme0n1
    model: INTEL NVMETHING
    name: nvme0n1
    serial: 1234567890ABCEF
    type: disk
    wipe: superblock
  [... stuff ...]
  - fstype: ext4
    id: nvme0n1_format
    label: ''
    type: format
    uuid: 32bb9415-c41b-41ed-843c-075e9bf344b9
    volume: nvme0n1
  [... stuff ...]

And in your own preseed you’d have :

storage:
  config:
  - id: nvme0n1
    model: INTEL NVMETHING
    name: nvme0n1
    serial: 1234567890ABCEF
    type: disk
    wipe: superblock
  [... stuff ...]
  - id: nvme0n1_crypt
    type: dm_crypt
    dm_name: nvme0n1_crypt
    volume: nvme0n1
    key: tempkey
    keysize: '512'
  - fstype: ext4
    id: nvme0n1_format
    label: ''
    type: format
    uuid: 32bb9415-c41b-41ed-843c-075e9bf344b9
    volume: nvme0n1_crypt
  [... stuff ...]

Again, the disk block didn’t change. I added the dm_crypt block, and updated the format block from volume: nvme0n1 to volume: nvme0n1_crypt.

Once you’re done with the storage section, we still need to add some base stanzas to it so that it’s complete.

At the very top, add the following :

#cloud-config
debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  maas: [wget, '--no-proxy', {{node_disable_pxe_url|escape.json}}, '--post-data', {{node_disable_pxe_data|escape.json}}, '-O', '/dev/null']

And now your preseed is complete !

LUKS2

At the time of this writing, curtin cannot create LUKS2 volumes directly, it can only do LUKS1. However, it’s possible to convert a LUKS1 volume to a LUKS2 volume, provided that the volume isn’t in use. Since / is pretty much always in use, it’s not possible
to do that on a running system ; you’d need to reboot in rescue mode and do it from there, which can be a pain, above all at scale.

But don’t despair, as it’s possible to do with during the deployment of your server ! You just add some commands in the late_commands block in your preseed. These commands will :

  • umount all the filesystems
  • convert the LUKS1 volume to LUKS2
  • update fstab to look for LUKS2 volumes

Below is an example for a system with the following storage layout :

  • /dev/sda2 is /boot, unencrypted
  • /dev/sda3 is /, encrypted
  • /dev/sdb2 is /srv/backups, encrypted
  • /dev/nvme0n1 is /srv/application, encrypted

You’d use the following late_commands block :

late_commands:
  00_fix_fstab: ['curtin', 'in-target', '--', 'perl', '-p', '-i', '-e', 's/LUKS1/LUKS2/', '/etc/fstab']
  01_unmount_sdb: ['umount', '/dev/mapper/sdb2_crypt']
  01_unmount_nvme0n1: ['umount', '/dev/mapper/nvme0n1_crypt']
  01_unmount_boot: ['umount', '/dev/sda2']
  02_unmount_sda: ['umount', '/dev/mapper/sda3_crypt']
  10_luksClose_sda: ['cryptsetup', 'luksClose', 'sda3_crypt']
  10_luksClose_sdb: ['cryptsetup', 'luksClose', 'sdb2_crypt']
  10_luksClose_nvme0n1: ['cryptsetup', 'luksClose', 'nvme0n1_crypt']
  20_luks2_sda: ['cryptsetup', '--batch-mode', 'convert', '/dev/sda3', '--type', 'luks2']
  20_luks2_sdb: ['cryptsetup', '--batch-mode', 'convert', '/dev/sdb2', '--type', 'luks2']
  20_luks2_nvme0n1: ['cryptsetup', '--batch-mode', 'convert', '/dev/nvme0n1', '--type', 'luks2']
  maas: [wget, '--no-proxy', {{node_disable_pxe_url|escape.json}}, '--post-data', {{node_disable_pxe_data|escape.json}}, '-O', '/dev/null']

So as you can see, we start by updating fstab, then we unmount everyhing (don’t forget to unmount
the non-encrypted partitions). / must be unmounted last !
Then we actually convert to LUKS2 and we’re done.

Install the preseed

Now you have your preseed, you just need to put it in the right place and with the right name.

If your MAAS is installed via an Ubuntu package, the directory is
/etc/maas/preseeds/. If it’s installed via a snap, it’s /var/snap/maas/current/preseeds/.

Preseed file names are important, as MAAS infers the scope of the preseed from its file name.

For example, if you want to apply the preseed on a single amd64 node named testnode.maas in your MAAS for any 20.04 deploy, you’d name the file :
curtin_userdata_ubuntu_amd64_generic_focal_testnode

If you want to apply it to all amd64 machines, you’d name it :
curtin_userdata_ubuntu_amd64.

Here is more documentation about preseeds if you need it.

So pick the appropriate name, and put the file in the appropriate directory.

Deploy and profit

Now that the preseed is in place, just deploy your server and MAAS should do the right thing ! Upon reboot, you’ll be asked to enter the encryption key, which is set to tempkey if you used the examples above.

The first thing you should do after the deploy is change the key
on ALL encrypted devices, using :

$ sudo cryptsetup luksChangeKey /dev/<device>

for each device. This will also update the key slot to the LUKS2
standards if you have opted to convert your LUKS volume to LUKS2.

Thanks for reading, hopefully this will be useful.
Feel free to ask any question :slight_smile:

Cheers

Resources :

5 Likes

Outstanding, @axino. Thank you for posting this.

Holy cruise ship! Book marking this.

Fantastic!

1 Like

@billwear - any way to help improve this workflow? Perhaps a CLI arg here that could define an OS, and thus would allow for the curtin config to be retrieved - without needing to first “deploy” in a placebo sense

1 Like

@knaledge, good question! i dunno, i’ll have to play around with it when i have time.

1 Like

This is a great post. Thanks for sharing!

The final curtin-config will work for all machines irrespective of the tags assosciated. Is there any way to give a particular disk layout for machines with a specific MAAS tag?

2 Likes

Wow, really nice @axino.
Impressive work! Looking forward to testing this because the missing feature of full disk encryption has been keeping me away from using MAAS in production.
Maybe finally I can make the switch to MAAS.

2 Likes