Simple custom storage RAID layout fails deploy stage with Error "Mount the root '/' filesystem to be able to deploy this node"

I’m testing out MAAS and trying to deploy a VM node using a custom storage layout. The VM node has 2 x 16GB disks, and in this test, I’m trying to create a simple RAID 1 across both the disks and mount that as the root filesystem.
Here is the commissioning script:

#!/bin/bash
# --- Start MAAS 1.0 script metadata ---
# name: 43-zoox-simple-raid-test
# title: Configure TCS node storage layout
# description: Configure the disk layout for the OS
# script_type: commissioning
# timeout: 5
# --- End MAAS 1.0 script metadata ---

cat > "$MAAS_STORAGE_CONFIG_FILE" <<EOL
{
  "layout": {
    "storage": {
      "type": "raid",
      "level": 1,
      "members": [
        "sda",
        "sdb"
      ],
      "fs": "ext4"
    }
  },
  "mounts": {
    "/": {
      "device": "storage"
    }
  }
}
EOL

I uploaded that script, then set the default storage layout to “Custom”. I then started the Commissioning process for the node. The node successfully completes all commissioning steps, including the script I uploaded.
I looked at the output of the 50-maas-01-commissioning script and saw that the storage-extra section had all the settings from my script:

  "storage-extra": {
    "layout": {
      "storage": {
        "type": "raid",
        "level": 1,
        "members": [
          "sda",
          "sdb"
        ],
        "fs": "ext4"
      }
    },
    "mounts": {
      "/": {
        "device": "storage"
      }
    }
  }

However, when I try to Deploy the node, it immediately throws an error that says Error: Mount the root '/' filesystem to be able to deploy this node

I see the following traceback in /var/snap/maas/common/log/regiond.log:

2023-06-02 22:49:54 maasserver.preseed: [warn] WARNING: '/snap/maas/27405/etc/maas/preseeds/curtin_userdata' contains deprecated preseed variables. Please remove: main_archive_directory, ports_archive_directory
2023-06-02 22:49:54 maasserver.websockets.protocol: [critical] Error on request (12) machine.action: {"storage": ["Mount the root '/' filesystem to be able to deploy this node."]}
	Traceback (most recent call last):
	  File "/usr/lib/python3.10/threading.py", line 953, in run
	    self._target(*self._args, **self._kwargs)
	  File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 822, in worker
	    return target()
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
	    task()
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
	    task()
	--- <exception caught here> ---
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
	    result = inContext.theWork()  # type: ignore[attr-defined]
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
	    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
	    return func(*args, **kw)
	  File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 857, in callInContext
	    return func(*args, **kwargs)
	  File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
	    result = func(*args, **kwargs)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/utils/orm.py", line 771, in call_within_transaction
	    return func_outside_txn(*args, **kwargs)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/utils/orm.py", line 574, in retrier
	    return func(*args, **kwargs)
	  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
	    return func(*args, **kwds)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/websockets/base.py", line 437, in prep_user_execute
	    return self._call_method_track_queries(
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/websockets/base.py", line 461, in _call_method_track_queries
	    result = method(params)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/websockets/handlers/machine.py", line 1089, in action
	    success_count, failed_system_ids = self._bulk_action(
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/websockets/handlers/machine.py", line 1056, in _bulk_action
	    self._action(machine, action_name, extra_params)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/websockets/handlers/machine.py", line 1043, in _action
	    return action.execute(**extra_params)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/node_action.py", line 186, in execute
	    self._execute(*args, **kwargs)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/node_action.py", line 589, in _execute
	    self.node.start(
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/utils/orm.py", line 755, in call_within_transaction
	    return func_within_txn(*args, **kwargs)
	  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
	    return func(*args, **kwds)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/models/node.py", line 5601, in start
	    return self._start(
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/utils/orm.py", line 755, in call_within_transaction
	    return func_within_txn(*args, **kwargs)
	  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
	    return func(*args, **kwds)
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/models/node.py", line 5781, in _start
	    self._start_deployment()
	  File "/snap/maas/27405/lib/python3.10/site-packages/maasserver/models/node.py", line 1772, in _start_deployment
	    raise ValidationError({"storage": storage_layout_issues})
	django.core.exceptions.ValidationError: {'storage': ["Mount the root '/' filesystem to be able to deploy this node."]}

Is there some incapability of the layout to mount / on a RAID device?

I have verified that a couple of other basic custom disk configurations (that just create and mount partitions with no RAID) work fine.
Does anyone know what the issue could be?

1 Like

I was hoping that there was just a compatibility problem and that I could just upgrade my MAAS server from Ubuntu 20.04 to Ubuntu 22.04, but no dice. I uninstalled maas, upgraded to 22.04, re-installed maas via the apt install method and setup MAAS again from scratch. I’m still getting the same error when deploying using this storage layout.

Looking through the /var/log/maas/rsyslog/.../messages log, I found the following interesting error:

udisksd[1105]: failed to load module mdraid: libbd_mdraid.so.2: cannot open shared object file: No such file or
 directory

followed shortly after by this error:

udisksd[1105]: Failed to load the 'mdraid' libblockdev plugin

I don’t know how MAAS does the RAID setup, but I would venture to guess that it needs mdraid. But this might be completely irrelevant if the actual disk partitioning and formatting is done at the Deploy stage, as opposed to commissioning.

I was able, after some testing, to verify that the actual disk partitioning and formatting is done during the deploy stage, so this error must have to do specifically with the way the config file is laid out.

Coming back to the problem with a fresh pair of eyes, I realized that the main error of consequence in the Python traceback was the ValidationError:

raise ValidationError({"storage": storage_layout_issues})

Since I couldn’t find any information in the logs about what part of the config was considered invalid, I could only assume that the layout file doesn’t support a simple full-disk RAID layout, and that you might have to specify a bootable partition (e.g. /boot/efi).

I created a new storage commissioning script, this time with a bootable partition:

#!/bin/bash
# --- Start MAAS 1.0 script metadata ---
# name: 43-zoox-simple-raid-test
# title: Configure TCS node storage layout
# description: Configure the disk layout for the OS
# script_type: commissioning
# timeout: 10
# --- End MAAS 1.0 script metadata ---

cat > "$MAAS_STORAGE_CONFIG_FILE" <<EOL
{
    "layout": {
        "sda": {
            "type": "disk",
            "ptable": "gpt",
            "partitions": [
                {
                    "name": "sda1",
                    "bootable": true,
                    "fs": "fat32",
                    "size": "1G"
                },
                {
                    "name": "sda2",
                    "fs": "ext4",
                    "size": "10G"
                },
                {
                    "name": "sda3",
                    "size": "5G"
                }
            ]
        },
        "sdb": {
            "type": "disk",
            "ptable": "gpt",
            "partitions": [
                {
                    "name": "sdb1",
                    "fs": "fat32",
                    "size": "1G"
                },
                {
                    "name": "sdb2",
                    "fs": "ext4",
                    "size": "10G"
                },
                {
                    "name": "sdb3",
                    "size": "5G"
                }
            ]
        },
        "md0": {
            "type": "raid",
            "level": 1,
            "members": [
                "sda3",
                "sdb3"
            ],
            "fs": "ext4"
        }
    },
    "mounts": {
        "/mnt": {
            "device": "md0"
        },
        "/boot/efi": {
            "device": "sda1"
        },
        "/": {
            "device": "sda2"
        }
    }
}
EOL

Once I uploaded this script and re-commissoned the node, the “Storage” tab of the node showed me the layout I expected:


Then I was able to Deploy the node without any errors. The layout looked exactly as I expected.

1 Like

I’ll also mention here for posterity that I was also able to mount the RAID device to / once I added a “bootable” partition

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.