Failing to achieve symmetric Linux RAID1 for root file system on top of 2 identical SSD devices


#1

NOTE: I don’t use UEFI, neither GPT.

Literally all my production-ready physical Machines have RAID1 for its root file system, in a layout that it’s for sure, the simplest possible.

Also, MOST IMPORTANT, both devices are bootable! If /dev/sda dies, I can easily boot into /dev/sdb by just removing /dev/sda and that’s it.

With MaaS, this looks “impossible” to do, I’ll try to exemplify here…

What I do manually (1), what I do via MaaS (2) and what I’m getting out of it (3)? As follows:

1- When I manually install Ubuntu 18.04 or Debian 9 in a machine with 2 identical storage devices, here is the end result:

root@server-1:~# sfdisk -l /dev/sda
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x4ffe919d

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1  *     2048 468860927 468858880 223.6G fd Linux raid autodetect


root@server-1:~# sfdisk -l /dev/sdb
Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x05877edb

Device     Boot Start       End   Sectors   Size Id Type
/dev/sdb1        2048 468862127 468860080 223.6G fd Linux raid autodetect

See how simple it’s? Two SSDs, 1 primary parition on each and NOTHING more.

Now, the Linux RAID1:

root@server-1:~# cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[2]
      234297344 blocks super 1.2 [2/2] [UU]
      bitmap: 2/2 pages [8KB], 65536KB chunk

unused devices: <none>

And the root file system:

root@server-1:~# mount | grep md0
/dev/md0 on / type ext4 (rw,relatime,errors=remount-ro)

NOTE: The /boot is inside of the RAID1 array, Grub works just fine. The only detail, that I can see during Ubuntu/Debian install, is that the installation process is smart enough to install Grub on both /dev/sda and on /dev/sdb, but, just to make sure, I run after first boot:

grub-install --recheck /dev/sda
grub-install --recheck /dev/sdb

So far, so good!

Now, when I tried to create the SAME SETUP as above, using MaaS, it fails to do it! The interface MaaS UI even shows layout X but, the deployed result is different!!!

2- What I’m doing via MaaS:

Screenshot:

First problem:

MaaS is using MBR for /dev/sda and GPT for /dev/sdb, only God knows why… lol

Second problem, MaaS is a liar :stuck_out_tongue: ! There is another partition there, created by MaaS, that doesn’t show up on UI!

Third problem, the partition type isn’t RAID Autodetect.

Let me show:

root@maas-deployed-server-1:~# sfdisk -l /dev/sda
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0a13ea3c

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1        2048 468846591 468844544 223.6G 83 Linux


root@maas-deployed-server-1:~# sfdisk -l /dev/sdb
Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2B89AF6F-5348-4E6F-B6A7-0165F4503EF8

Device     Start       End   Sectors   Size Type
/dev/sdb1   2048      4095      2048     1M BIOS boot
/dev/sdb2   4096 468848639 468844544 223.6G Linux filesystem

I wasn’t expecting MaaS to create /dev/sdb1 with “BIOS boot” and this partition doesn’t show up on MaaS UI!

Two identical disks, asked for RAID1 but, MaaS failed to deliver what I asked.

root@maas-deployed-server-1:~# cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb2[1] sda1[0]
      234290176 blocks super 1.2 [2/2] [UU]
      bitmap: 1/2 pages [4KB], 65536KB chunk

unused devices: <none>

MaaS is using /dev/sda1 and /dev/sdb2

So, bottom line is: How to fix this?

I’m starting to look for custom partitioning, like:

http://caribou.kamikamamak.com/2015/06/26/custom-partitioning-with-maas-and-curtin-2/

But I would rather choose something simpler, like I do with manual setup. I think that I can translate the Ubuntu/Debian manual setup into some simple code for Preseed, using Debian Installer…

Again, I would prefer something KISS.

To fix this manually, since the problem with first SSD is only the type of the partition, I can manually re-do the /dev/sdb, but it would be AWESOME if MaaS could do what I do manually!

Including an option to select 2 Boot Disks! So Grub will end up being installed on two devices identical devices.

NOTE: I didn’t tried to boot this /dev/sdb created by MaaS, is it booting? I’m not sure yet, anyway, I would rather keep the two identical disks with the very same scheme / layout anyway.

I appreciate any help!

I love MaaS ideas but I must confess that the implementation is… How can I say… A bit unstable.

Cheers!
Thiago


#2

I guess that it’s time to try this!

:smiley:

But, what about RAID1? I know almost nothing about ZFS… :man_facepalming:

And boot devices being bootable!

I’m kinda of getting tired of mdadm and ext4… lol

Tried btrfs on root, works great! But only good for LXD Containers… Not good to host QCoW2 files.

So, ZFS root it is!

But I’m also looking for compress and dedup outside of ZFS, so, maybe, ZFS root on top of VDO? Or 2 VDO’s, one for each disk, then ZFS Mirror? I’m confused… LOL

I’ll start small, I’ll try ZFS Root via MaaS first in a Pod VM, to see how everything fits together, then, I’ll try to install Ubuntu 18.04 on top of ZFS by myself (no MaaS), then, raid1. For now, no VDO, no DRBD, just 1 /dev/nvme0n1 for ZFS root, .

:open_mouth:


#3

A few questions:

  1. Version of curtin in use? (e.g. dpkg -l | grep curtin)
  2. What’s the resulting configuration of your RAID: maas machine get-curtin-config
  3. Exact version of MAAS ?

#4

Hi !
The tip here (not sure it is the normal behaviour) is to select the checkbox “Boot” on the disk you want to create a partition.

  • You check “boot” for sda and you create your partition sda1 on it

Then if you let “boot” on sda and create sdb parition it will create sdb2 with GPT and it is not what you want

  • So at this moment check “boot” on sdb disk and create your new partition that will be sdb1 using MBR !

Ludwig


#5

Thank you so much ludww! Your workaround worked like a charm! :smiley:

I finally have a MaaS deployed Machine with RAID1 and still using BIOS / MBR.

I’m feeling that the MaaS UI could be improved here… :wink: