Deploying servers with full disk encryption (LUKS2)

Hi everybody,

I recently had to deploy a few “full disk encrypted” servers with MAAS. While this is not something that’s supported officially, it’s relatively easy to do with preseeds. And once it’s done, you can scale out pretty easily.

An overview of the steps to follow are :

  1. get the server in MAAS and configure it
  2. fetch the curtin config
  3. create the preseed
  4. deploy

Requirements

You will require functional MAAS CLI. See https://maas.io/docs/maas-cli for more info.

Enlist, commission, configure

First, enlist and commission your server, and then configure its storage and networking as normal.

Fetch the curtin config

We’re now going to grab the curtin config that MAAS generates for our server. Since this depends on the OS you deploy, MAAS won’t give you said config unless the server is in a “deployment” state. So start a deploy of the target OS,
and then run the following from the MAAS CLI :

$ maas <profile> machine get-curtin-config <machine-id> > /tmp/curtin-config

You can now cancel the deploy.

IMPORTANT : if you already have a preseed matching your server deployment, the command above will return your preseed and not the MAAS-generated curtin config. Make sure that you do not have a preseed matching your server deployment when running the command above.

Create the preseed

Preseeds offer you a way to override parts of the default installation process. In this case, we’re going to rework the storage setup part.

Note that a preseed completely overwrites the MAAS-generated curtin config, so it needs to be “complete”. If you only put a storage: block in your preseed, the MAAS deploy will fail.

Open /tmp/curtin-config in an editor. Keep only the block starting by storage:. You’re going to recognize the storage you configured in MAAS. This YAML list can be a bit frightening at the beginning, but if you take a close look you’ll notice
that the list is sorted on the type key.

  • First, type: disk. This sets up disks for use by curtin.
  • Then, type: partition. This creates partitions on the disks.
  • Then, type: format. This formats the partitions with the specified filesystem.
  • And finally, type: mount. This mounts the filesystem on a mountpoint.

For the sake of this example, I’m going to assume that each LUKS device will only have one filesystem on it. curtin can create LUKS volumes using the dm_crypt type. We’re going to insert this between the partition (or disk if you want
to encrypt a full disk) and format stanzas.

For each partition block, you’re going to add a dm_crypt stanza, and then modify the corresponding format stanza to reference the encrypted partition.

Let’s take an example. Assume you have a /dev/sda3 partition that you want to encrypt. MAAS would likely have generated something like that :

storage:
  config:
  [... some disk/partitions stanzas ...]
  - device: sda
    id: sda-part3
    name: sda-part3
    number: 3
    size: 599085023232B
    type: partition
    uuid: 10a1b6a4-f31c-4502-abb0-537ba788fd2a
    wipe: superblock
  [... some more stuff possibly ...]
  - fstype: ext4
    id: sda-part3_format
    label: ''
    type: format
    uuid: 2ae7cb24-fcd4-4985-8f7b-7ed1752ccd2f
    volume: sda-part3
  [... even more stuff maybe ...]

Right. So as per above, we’re going to add a dm_crypt entry between these two, and modify the format stanza to refer to the dm_crypt entry. This would give ;

storage:
  config:
  [... some disk/partitions stanzas ...]
  - device: sda
    id: sda-part3
    name: sda-part3
    number: 3
    size: 599085023232B
    type: partition
    uuid: 10a1b6a4-f31c-4502-abb0-537ba788fd2a
    wipe: superblock
  [... some more stuff possibly ...]
  - id: sda-part3_crypt
    type: dm_crypt
    dm_name: sda3_crypt
    volume: sda-part3
    key: tempkey
    keysize: '512'
  - fstype: ext4
    id: sda-part3_format
    label: ''
    type: format
    uuid: 2ae7cb24-fcd4-4985-8f7b-7ed1752ccd2f
    volume: sda-part3_crypt
  [... even more stuff maybe ...]

I didn’t touch the partition block. I added the dm_crypt block, which is referring to the partition item via volume: sda-part3. And then I updated the format block, just replacing volume: sda-part3 by volume: sda-part3_crypt.

If you want to encrypt a full disk, it’s pretty much the same.
MAAS-generated curtin config would be :

storage:
  config:
  - id: nvme0n1
    model: INTEL NVMETHING
    name: nvme0n1
    serial: 1234567890ABCEF
    type: disk
    wipe: superblock
  [... stuff ...]
  - fstype: ext4
    id: nvme0n1_format
    label: ''
    type: format
    uuid: 32bb9415-c41b-41ed-843c-075e9bf344b9
    volume: nvme0n1
  [... stuff ...]

And in your own preseed you’d have :

storage:
  config:
  - id: nvme0n1
    model: INTEL NVMETHING
    name: nvme0n1
    serial: 1234567890ABCEF
    type: disk
    wipe: superblock
  [... stuff ...]
  - id: nvme0n1_crypt
    type: dm_crypt
    dm_name: nvme0n1_crypt
    volume: nvme0n1
    key: tempkey
    keysize: '512'
  - fstype: ext4
    id: nvme0n1_format
    label: ''
    type: format
    uuid: 32bb9415-c41b-41ed-843c-075e9bf344b9
    volume: nvme0n1_crypt
  [... stuff ...]

Again, the disk block didn’t change. I added the dm_crypt block, and updated the format block from volume: nvme0n1 to volume: nvme0n1_crypt.

Once you’re done with the storage section, we still need to add some base stanzas to it so that it’s complete.

At the very top, add the following :

#cloud-config
debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  maas: [wget, '--no-proxy', {{node_disable_pxe_url|escape.json}}, '--post-data', {{node_disable_pxe_data|escape.json}}, '-O', '/dev/null']

And now your preseed is complete !

LUKS2

At the time of this writing, curtin cannot create LUKS2 volumes directly, it can only do LUKS1. However, it’s possible to convert a LUKS1 volume to a LUKS2 volume, provided that the volume isn’t in use. Since / is pretty much always in use, it’s not possible
to do that on a running system ; you’d need to reboot in rescue mode and do it from there, which can be a pain, above all at scale.

But don’t despair, as it’s possible to do with during the deployment of your server ! You just add some commands in the late_commands block in your preseed. These commands will :

  • umount all the filesystems
  • convert the LUKS1 volume to LUKS2
  • update fstab to look for LUKS2 volumes

Below is an example for a system with the following storage layout :

  • /dev/sda2 is /boot, unencrypted
  • /dev/sda3 is /, encrypted
  • /dev/sdb2 is /srv/backups, encrypted
  • /dev/nvme0n1 is /srv/application, encrypted

You’d use the following late_commands block :

late_commands:
  00_fix_fstab: ['curtin', 'in-target', '--', 'perl', '-p', '-i', '-e', 's/LUKS1/LUKS2/', '/etc/fstab']
  01_unmount_sdb: ['umount', '/dev/mapper/sdb2_crypt']
  01_unmount_nvme0n1: ['umount', '/dev/mapper/nvme0n1_crypt']
  01_unmount_boot: ['umount', '/dev/sda2']
  02_unmount_sda: ['umount', '/dev/mapper/sda3_crypt']
  10_luksClose_sda: ['cryptsetup', 'luksClose', 'sda3_crypt']
  10_luksClose_sdb: ['cryptsetup', 'luksClose', 'sdb2_crypt']
  10_luksClose_nvme0n1: ['cryptsetup', 'luksClose', 'nvme0n1_crypt']
  20_luks2_sda: ['cryptsetup', '--batch-mode', 'convert', '/dev/sda3', '--type', 'luks2']
  20_luks2_sdb: ['cryptsetup', '--batch-mode', 'convert', '/dev/sdb2', '--type', 'luks2']
  20_luks2_nvme0n1: ['cryptsetup', '--batch-mode', 'convert', '/dev/nvme0n1', '--type', 'luks2']
  maas: [wget, '--no-proxy', {{node_disable_pxe_url|escape.json}}, '--post-data', {{node_disable_pxe_data|escape.json}}, '-O', '/dev/null']

So as you can see, we start by updating fstab, then we unmount everyhing (don’t forget to unmount
the non-encrypted partitions). / must be unmounted last !
Then we actually convert to LUKS2 and we’re done.

Install the preseed

Now you have your preseed, you just need to put it in the right place and with the right name.

If your MAAS is installed via an Ubuntu package, the directory is
/etc/maas/preseeds/. If it’s installed via a snap, it’s /var/snap/maas/current/preseeds/.

Preseed file names are important, as MAAS infers the scope of the preseed from its file name.

For example, if you want to apply the preseed on a single amd64 node named testnode.maas in your MAAS for any 20.04 deploy, you’d name the file :
curtin_userdata_ubuntu_amd64_generic_focal_testnode

If you want to apply it to all amd64 machines, you’d name it :
curtin_userdata_ubuntu_amd64.

Here is more documentation about preseeds if you need it.

So pick the appropriate name, and put the file in the appropriate directory.

Deploy and profit

Now that the preseed is in place, just deploy your server and MAAS should do the right thing ! Upon reboot, you’ll be asked to enter the encryption key, which is set to tempkey if you used the examples above.

The first thing you should do after the deploy is change the key
on ALL encrypted devices, using :

$ sudo cryptsetup luksChangeKey /dev/<device>

for each device. This will also update the key slot to the LUKS2
standards if you have opted to convert your LUKS volume to LUKS2.

Thanks for reading, hopefully this will be useful.
Feel free to ask any question :slight_smile:

Cheers

Resources :

9 Likes

Outstanding, @axino. Thank you for posting this.

Holy cruise ship! Book marking this.

Fantastic!

1 Like

@billwear - any way to help improve this workflow? Perhaps a CLI arg here that could define an OS, and thus would allow for the curtin config to be retrieved - without needing to first “deploy” in a placebo sense

1 Like

@knaledge, good question! i dunno, i’ll have to play around with it when i have time.

1 Like

This is a great post. Thanks for sharing!

The final curtin-config will work for all machines irrespective of the tags assosciated. Is there any way to give a particular disk layout for machines with a specific MAAS tag?

2 Likes

Wow, really nice @axino.
Impressive work! Looking forward to testing this because the missing feature of full disk encryption has been keeping me away from using MAAS in production.
Maybe finally I can make the switch to MAAS.

2 Likes

@axino that is fantastic. I wonder why full disk encryption is not supported.

I was considering to use cryptsetup-reencrypt at a last step in the provisioning, but I amnot sure if it would work. See if there is a generic way of turning an LVM install into a LUKS-LVM install.

theoretically you can encrypting “in-place” a filesystem that is not yet encrypted.

Hi @axino, thanks for this post! this is what we are looking for recently.

I tried this method on provisioning VMs, but the system won’t boot, says it couldn’t find the boot device, followed by the old UUID inherited from the preseed file. also tried removed the UUID lines in the preseed file doesn’t work either (same error). curtin installation log of that build shows it finished without errors but deployment failed. Are there anything that I could have missed?

Hi @gaoshan - and welcome !

it’s hard to say without more details. Did you keep an unencrypted /boot ? I think I’d need the full preseed and the install logs to troubleshoot further.

Cheers

1 Like

Thanks for the quick reply @axino. Yes, I only encrypted the ‘/’ , but not ‘/boot’, below is the preseed file content – (paste the entire installation log here takes so much space, but let me know if you’d like to see it, and I will find a way to share it with you. Thanks again!)

#cloud-config
debconf_selections:
maas: |
{{for line in str(curtin_preseed).splitlines()}}
{{line}}
{{endfor}}

late_commands:
maas: [wget, ‘–no-proxy’, {{node_disable_pxe_url|escape.json}}, ‘–post-data’, {{node_disable_pxe_data|escape.json}}, ‘-O’, ‘/dev/null’]

storage:
config:

  • grub_device: true
    id: sda
    name: sda
    path: /dev/sda
    ptable: gpt
    type: disk
    wipe: superblock
  • device: sda
    flag: bios_grub
    id: sda-part1
    number: 1
    offset: 4194304B
    size: 1048576B
    type: partition
    wipe: zero
  • device: sda
    id: sda-part2
    name: sda-part2
    number: 2
    size: 64416120832B
    type: partition
    uuid: d1980a81-6dc3-4394-80d6-b8c60042358b
    wipe: superblock
  • id: sda-part2_crypt
    type: dm_crypt
    dm_name: sda2_crypt
    volume: sda-part2
    key: tempkey
    keysize: ‘512’
  • fstype: ext4
    id: sda-part2_format
    label: root
    type: format
    uuid: dae49194-33a5-42c8-b4a7-2fdef442f873
    volume: sda-part2_crypt
  • device: sda-part2_format
    id: sda-part2_mount
    path: /
    type: mount
    version: 1
    verbosity: 3

One more thing to add: I booted the VM using a live cd of ubuntu, then mount the partition where OS was installed, “sudo mount /dev/sda2 /home/ubuntu/temp”, got an error says "unknown filesystem type 'crypto_LUKS’,so looks like the volume got encrypted, but somehow the system doesn’t recognize the filesystem. BTW, the last deployment reboot did not ask me to type in the default key which is ‘tempkey’ to boot up the server. and get into ‘grub rescue’ mode right away with two errors:

  1. error: no such device: dae49194-33a5-42c8-b4a7-2fdef442f873
  2. error: unknown filesystem

From your preseed, I guess you want to use sda1 as /boot. In which case, you’re missing 2 stanzas : one stanza of type “format” to format sda1 (presumably with ext4), and one stanza of type “mount” to mount sda1 under /boot.

Did you add an ext4 /boot partition under MAAS web interface ? Because if you did, these stanzas should have appeared in the curtin config that MAAS gave you.

One last thing : if you’re using EFI to boot your VM, you will also need a separate partition for /boot/efi, using the fat32 filesystem.

1 Like

Thanks @axino. I will try it out.

Did you add an ext4 /boot partition under MAAS web interface ? Because if you did, these stanzas should have appeared in the curtin config that MAAS gave you. ------ No, what I use for now on MAAS storage is default which I believe is ‘flat layout’ .

Thanks @axino. partition encryption works now.

I tried full disk encryption as well, but looks like I couldn’t find a “format” stanza for disk itself in MAAS generated config templates (I added one for the disk, but deployment fails) , it’s always for partitions. What storage layout should be used when do full disk encryption?

I think the term “full disk encryption” is generally accepted to mean "everything is encrypted apart from /boot". There are ways to encrypt /boot, see for example this page (found after a very quick search).

If you’re talking about extra disks (where there’s no /boot) then I don’t know if curtin supports using a volume that’s a full disk in a type: dm-crypt stanza. If not, you can just create one partition that spans the whole disk.