Commission machines (snap/2.7/CLI)

MAAS is built to manage machines, including the operating systems on those machines. Enlistment and commissioning are features that make it easier to start managing a machine – as long as that machine has been configured to netboot. Enlistment enables users to simply connect a machine, configure the firmware properly, and power it on so that MAAS can find it and add it.

Twelve questions you may have:

  1. Tell me about enlistment vs. commissioning.
  2. How are machines commissioned?
  3. How can I commission NUMA and SR-IOV nodes?
  4. What are MAAS commissioning scripts?
  5. What post-commission configuration is possible?
  6. What is a bond interface and how do I create one?
  7. What is a bridge interface and how do I create one?
  8. How do I assign an interface to a fabric?
  9. How do I discover interface identifiers?
  10. How do I delete an interface?
  11. How do I create a VLAN interface?
  12. How do I delete a VLAN interface?

Enlistment versus commissioning

Enlistment happens when MAAS starts; it reaches out on connected subnets to locate any nodes – that is, devices and machines – that reside on those subnets. MAAS finds a machine that’s configured to netboot (e.g., via PXE), boots that machine into Ubuntu, and then sends cloud-init user data which runs standard (i.e., built-in) commissioning scripts. The machine actually adds itself over the MAAS API, and then requests permission to send commissioning data.

Since MAAS doesn’t know whether you might intend to actually include these discovered machines in your cloud configuration, it won’t automatically take them over, but it will read them to get an idea how they’re set up. MAAS then presents these machines to you with a MAAS state of “New.” This allows you to examine them and decide whether or not you want MAAS to manage them.

When you configure a machine to netboot – and turn it on while connected to the network – MAAS will enlist it, giving it a status of “New.” You can also add a machine manually). In either case, the next step is commissioning, which boots the machine into an ephemeral Ubuntu kernel so that resource information can be gathered. This resource information includes (but isn’t limited to) information about CPUs, memory, and storage. You can also run custom commissioning scripts to meet your specific needs.

How machines are commissioned

When MAAS commissions a machine, the following sequence of events takes place:

  1. DHCP server is contacted
  2. kernel and initrd are received over TFTP
  3. machine boots
  4. initrd mounts a Squashfs image ephemerally over HTTP
  5. cloud-init runs built-in and custom commissioning scripts
  6. machine shuts down

The commissioning scripts will talk to the region API server to ensure that everything is in order and that eventual deployment will succeed.

MAAS chooses the latest Ubuntu LTS release as the default image for commissioning. If desired, you can select a different image in the “Settings” page of the web UI, by selecting the “General” tab and then scrolling down to the Commissioning section.

To commission a machine:

maas $PROFILE machine commission $SYSTEM_ID

To commission a node, it must have a status of “New”.

To commission all nodes in the “New” state:

maas $PROFILE machines accept-all

You have the option of setting some parameters to change how commissioning runs:

  1. enable_ssh: Optional integer. Controls whether to enable SSH for the commissioning environment using the user’s SSH key(s). ‘1’ == True, ‘0’ == False. Roughly equivalent to the Allow SSH access and prevent machine powering off in the web UI.

  2. skip_bmc_config: Optional integer. Controls whether to skip re-configuration of the BMC for IPMI based machines. ‘1’ == True, ‘0’ == False.

  3. skip_networking: Optional integer. Controls whether to skip re-configuring the networking on the machine after the commissioning has completed. ‘1’ == True, ‘0’ == False. Roughly equivalent to Retain network configuration in the web UI.

  4. skip_storage: Optional integer. Controls hether to skip re-configuring the storage on the machine after the commissioning has completed. ‘1’ == True, ‘0’ == False. Roughly equivalent to Retain storage configuration in the web UI.

  5. commissioning_scripts: Optional string. A comma separated list of commissioning script names and tags to be run. By default all custom commissioning scripts are run. Built-in commissioning scripts always run. Selecting update_firmware or configure_hba will run firmware updates or configure HBA’s on matching machines.

  6. testing_scripts: Optional string. A comma seperated list of testing script names and tags to be run. By default all tests tagged commissioning will be run. Set to none to disable running tests.

  7. parameters: Optional string. Scripts selected to run may define their own parameters. These parameters may be passed using the parameter name. Optionally a parameter may have the script name prepended to have that parameter only apply to that specific script.

Once commissioned, you may consider creating or applying a tag to this machine. The next step is deployment.

Commission NUMA and SR-IOV nodes

If you are using the NUMA architecture, MAAS versions 2.7 and higher guarantee that machines are assigned to a single NUMA node that contains all the machine’s resources. Node boundaries are critical, especially in vNUMA situations. Splitting nodes can create unnecessary latency. You want the NUMA node boundaries to match VM boundaries if at all possible.

You must recommission NUMA/SR-IOV machines that were previously commissioned under version 2.6 or earlier.

MAAS commissioning scripts

When a machine boots, MAAS first instructs it to run cloud-init to set up SSH keys (during commissioning only), set up NTP, and execute a script that runs other commissioning scripts. Currently, the sequence of MAAS-provided commissioning scripts proceeds like this:

  • 00-maas-00-support-info: MAAS gathers information that helps to identify and characterise the machine for debugging purposes, such as the kernel, versioning of various components, etc.

  • 00-maas-01-lshw: this script pulls system BIOS and vendor info, and generates user-defined tags for later use.

  • 00-maas-03-install-lldpd: this script installs the link layer discovery protocol (LLDP) daemon, which will later capture networking information about the machine. The lldpd needs to be installed early because it requires about a 60-second delay before running.

  • 00-maas-04-list-modaliases: this script figures out what hardware modules are loaded, providing a way to autorun certain scripts based on which modules are loaded.

  • 00-maas-05-dhcp-unconfigured-ifaces: MAAS will want to know all the ways the machine is connected to the network. Only PXE comes online during boot; this script brings all the other networks online so they can be recognised.

  • 00-maas-06-get-fruid-api-data: this script gathers information for the Facebook wedge power type.

  • 00-maas-08-serial-ports: this script lists what serial ports are available on the machine.

  • 40-maas-01-network-interfaces: this script is just used to get the IP address, which can then be associated with a VLAN/subnet.

  • 50-maas-01-commissioning: this script is the main MAAS tool, gathering information on machine resources, such as storage, network devices, CPU, RAM, etc. We currently pull this data using lxd: We use a Go binary built from lxd source that just contains the minimum source to gather the resource information we need. This script also checks whether the machine being commissioning is a virtual machine, which may affect how MAAS interacts with it.

  • 99-maas-01-capture-lldp: this script gathers LLDP network information to be presented on the logs page; this data is not used by MAAS at all.

  • 99-maas-05-kernel-cmdline: this script is used to update the boot devices; it double-checks that the right boot interface is selected.

Commissioning runs the same dozen or so scripts as enlistment, gathering all the same information, but with these two caveats:

  1. Commissioning also runs user-supplied commissioning scripts, if present. Be aware that these scripts run as root, so they can execute any system command.

  2. Commissioning runs test scripts which are not run during enlistment.

In both enlistment and commissioning, MAAS uses either the MAC address or the UUID to identify machines. Currently, because some machine types encountered by MAAS do not use unique MAC addresses, we are trending toward using the UUID.

Post-commission configuration

Once commissioned, you can configure the machine’s network interface(s). Specifically, when a machine’s status is either “Ready” or “Broken”, interfaces can be added/removed, attached to a fabric and linked to a subnet, and provided an IP assignment mode. Tags can also be assigned to specific network interfaces (see Tags for network interfaces).

If you want to edit the IP assignment mode of a network interface, the existing subnet link first needs to be removed.

Begin by finding the interface ID as well as the interface’s subnet link ID with the command:

maas $PROFILE node read $SYSTEM_ID

Once that’s done, proceed to unlink and link:

maas $PROFILE interface unlink-subnet $SYSTEM_ID $INTERFACE_ID id=$SUBNET_LINK_ID
maas $PROFILE interface link-subnet $SYSTEM_ID $INTERFACE_ID mode=$IP_MODE subnet=$SUBNET_CIDR [$OPTIONS]

For instance, to have interface 58, with subnet link 146, on machine exqn37 use DHCP on subnet 192.168.1.0/24:

maas $PROFILE interface unlink-subnet exqn37 58 id=146
maas $PROFILE interface link-subnet exqn37 58 mode=dhcp subnet=192.168.1.0/24

If instead of DHCP, you desire a static address, then the second command would look like this:

maas $PROFILE interface link-subnet exqn37 58 mode=static subnet=192.168.1.0/24 ip_address=192.168.1.113

See Concepts and terms for the definitions of reserved range types.

Bond interfaces and how to create one

A bond interface is used to aggregate two or more physical interfaces into a single logical interface. Combining multiple network connections in parallel can increase network throughput beyond what a single NIC will allow. It also provides some redundancy in case one of the NICs should fail. More information about the theory behind bonded NICs is found in the relevant IEEE standard.

A bond can be created with the following command:

maas $PROFILE interfaces create-bond $SYSTEM_ID name=$BOND_NAME \
parents=$IFACE1_ID mac_address=$MAC_ADDR \ 
parents=$IFACE2_ID bond_mode=$BOND_MODE \
bond_updelay=$BOND_UP bond_downdelay=$BOND_DOWN mtu=$MTU

Use the parents parameters to define which interfaces form the aggregate interface.

The bond_updelay and bond_downdelay parameters specify the number of milliseconds to wait before either enabling or disabling a slave after a failure has been detected.

The following is an example of create-bond in action:

maas admin interfaces create-bond 4efwb4 name=bond0 parents=4 \
mac_address=52:52:00:00:00:00 parents=15 bond_mode=802.3ad \
bond_updelay=200 bond_downdelay=200 mtu=9000

There are a wide range of bond parameters you can choose when creating a bond:

Parameter Type and description
mac_address Optional string. MAC address of the interface.
tags Optional string. Tags for the interface.
vlan Optional string. VLAN the interface is connected to. If not provided then the interface is considered disconnected.
parents Required integer. Parent interface ids that make this bond.
bond_miimon Optional integer. The link monitoring freqeuncy in milliseconds. (Default: 100).
bond_downdelay Optional integer. Specifies the time, in milliseconds, to wait before disabling a slave after a link failure has been detected.
bond_updelay Optional integer. Specifies the time, in milliseconds, to wait before enabling a slave after a link recovery has been detected.
bond_lacp_rate Optional string. Option specifying the rate at which to ask the link partner to transmit LACPDU packets in 802.3ad mode. Available options are fast or slow. (Default: slow).
bond_xmit_hash_policy Optional string. The transmit hash policy to use for slave selection in balance-xor, 802.3ad, and tlb modes. Possible values are: layer2, layer2+3, layer3+4, encap2+3, encap3+4. (Default: layer2)
bond_num_grat_arp Optional integer. The number of peer notifications (IPv4 ARP or IPv6 Neighbour Advertisements) to be issued after a failover. (Default: 1)
mtu Optional integer. Maximum transmission unit.
accept_ra Optional boolen. Accept router advertisements. (IPv6 only)
autoconf Optional boolean. Perform stateless autoconfiguration. (IPv6 only)
bond_mode Optional string. The operating mode of the bond. (Default: active-backup).

Supported bonding modes include:

Mode Behavior
balance-rr: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.
active-backup Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond’s MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.
balance-xor Transmit based on the selected transmit hash policy. The default policy is a simple [(source MAC address XOR’d with destination MAC address XOR packet type ID) modulo slave count].
broadcast Transmits everything on all slave interfaces. This mode provides fault tolerance.
802.3ad IEEE 802.3ad dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Uses all slaves in the active aggregator according to the 802.3ad specification.
balance-tlb Adaptive transmit load balancing: channel bonding that does not require any special switch support.
balance-alb Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation.

A bridge interface is created with the following syntax:

maas $PROFILE interfaces create-bridge $SYSTEM_ID name=$BRIDGE_NAME \
parent=$IFACE_ID

Use parent to define the primary interface used for the bridge:

maas admin interfaces create-bridge 4efwb4 name=bridged0 parent=4

The following parameters may be applied when creating a bridge:

  1. name: Optional string. Name of the interface.

  2. mac_address: Optional string. MAC address of the interface.

  3. tags: Optional string. Tags for the interface.

  4. vlan: Optional string. VLAN the interface is connected to.

  5. parent: Optional integer. Parent interface id for this bridge interface.

  6. bridge_type: Optional string. The type of bridge to create. Possible values are: standard, ovs.

  7. bridge_stp: Optional boolean. Turn spanning tree protocol on or off. (Default: False).

  8. bridge_fd: Optional integer. Set bridge forward delay to time seconds. (Default: 15).

  9. mtu: Optional integer. Maximum transmission unit.

  10. accept_ra: Optional boolean. Accept router advertisements. (IPv6 only)

  11. autoconf: Optional boolean. Perform stateless autoconfiguration. (IPv6 only)

Delete a bridge interface

The “delete” command can be used to delete a bridge interface, a bond interface or a physical interface:

maas $PROFILE interface delete $SYSTEM_ID $IFACE_ID

For example:

maas admin interface delete 4efwb4 15

The following is output after the successful deletion of an interface:

Success.
Machine-readable output follows:

Note that while the label is presented, there is no machine-readable output expected after the successful execution of the delete command.

Bridge interfaces and how to create one

A bridge interface is created with the following syntax:

maas $PROFILE interfaces create-bridge $SYSTEM_ID name=$BRIDGE_NAME \
parent=$IFACE_ID

Use ‘parent’ to define the primary interface used for the bridge:

maas admin interfaces create-bridge 4efwb4 name=bridged0 parent=4

Assign a network interface to a fabric

This task is made easier with the aid of the jq utility. It filters the maas command (JSON formatted) output and prints it in the desired way, which allows you to view and compare data quickly. Go ahead and install it:

sudo apt install jq

In summary, MAAS assigns an interface to a fabric by assigning it to a VLAN. First, we need to gather various bits of data.

List some information on all machines:

maas $PROFILE machines read | jq ".[] | \
    {hostname:.hostname, system_id: .system_id, status:.status}" --compact-output

Example output:

{"hostname":"machine1","system_id":"dfgnnd","status":4}
{"hostname":"machine2","system_id":"bkaf6e","status":6}
{"hostname":"machine4","system_id":"63wqky","status":6}
{"hostname":"machine3","system_id":"qwkmar","status":4}

You can only edit an interface when the corresponding machine has a status of ‘Ready’. This state is numerically denoted by the integer ‘4’.

List some information for all interfaces on the machine in question (identified by its system id ‘dfgnnd’):

maas $PROFILE interfaces read dfgnnd | jq ".[] | \
    {id:.id, name:.name, mac:.mac_address, vid:.vlan.vid, fabric:.vlan.fabric}" --compact-output

Example output:

{"id":8,"name":"eth0","mac":"52:54:00:01:01:01","vid":0,"fabric":"fabric-1"}
{"id":9,"name":"eth1","mac":"52:54:00:01:01:02","vid":null,"fabric":null}

List some information for all fabrics:

maas $PROFILE fabrics read | jq ".[] | \
    {name:.name, vlans:.vlans[] | {id:.id, vid:.vid}}" --compact-output

Example output:

{"name":"fabric-0","vlans":{"id":5001,"vid":0}}
{"name":"fabric-1","vlans":{"id":5002,"vid":0}}
{"name":"fabric-2","vlans":{"id":5003,"vid":0}}

This example will show how to move interface ‘8’ (on machine ‘dfgnnd’) from ‘fabric-1’ to ‘fabric-0’. Based on the gathered information, this will consist of changing the interface’s VLAN from ‘5002’ to ‘5001’:

maas $PROFILE interface update dfgnnd 8 vlan=5001 >/dev/null

Verify the operation by relisting information for the machine’s interface:

maas $PROFILE interfaces read dfgnnd | jq ".[] | \
    {id:.id, name:.name, mac:.mac_address, vid:.vlan.vid, fabric:.vlan.fabric}" --compact-output

The output shows that the interface is now on fabric-0:

{"id":8,"name":"eth0","mac":"52:54:00:01:01:01","vid":0,"fabric":"fabric-0"}
{"id":9,"name":"eth1","mac":"52:54:00:01:01:02","vid":null,"fabric":null}

Interface identifiers

The MAAS CLI uses a numeric interface identifier for many interface operations. Use the following command to retrieve the identifier(s):

maas $PROFILE interfaces read $SYSTEM_ID

Look for either id or the number at the end of an interface’s resource URI, such as 15 in the following example output:

"id": 15,
"mac_address": "52:54:00:55:06:40",
...
"name": "ens9",
...
"resource_uri": "/MAAS/api/2.0/nodes/4efwb4/interfaces/15/"

Delete an interface

The ‘delete’ command can be used to delete a bridge interface, a bond interface or a physical interface:

maas $PROFILE interface delete $SYSTEM_ID $IFACE_ID

For example:

maas admin interface delete 4efwb4 15

The following is output after the successful deletion of an interface:

Success.
Machine-readable output follows:

There is no machine-readable output after the successful execution of the delete command.

Create a VLAN interface

To create a VLAN interface, use the following syntax:

maas $PROFILE vlans create $FABRIC_ID name=$NAME vid=$VLAN_ID

For example, the following command creates a VLAN called 'Storage network:

maas admin vlans create 0 name="Storage network" vid=100

The above command generates the following output:

Success.
Machine-readable output follows:
{
    "vid": 100,
    "mtu": 1500,
    "dhcp_on": false,
    "external_dhcp": null,
    "relay_vlan": null,
    "name": "Storage network",
    "space": "undefined",
    "fabric": "fabric-0",
    "id": 5004,
    "primary_rack": null,
    "fabric_id": 0,
    "secondary_rack": null,
    "resource_uri": "/MAAS/api/2.0/vlans/5004/"
}

Be aware that the $VLAN_ID parameter does not indicate a VLAN ID that corresponds to the VLAN tag. You must first create the VLAN and then associate it with the interface:

maas $PROFILE interfaces create-vlan $SYSTEM_ID vlan=$OUTPUT_VLAN_ID \
parent=$IFACE_ID

OUTPUT_VLAN_ID corresponds to the id value output when MAAS created the VLAN.

The following example contains values that correspond to the output above:

maas admin interfaces create-vlan 4efwb4 vlan=5004 parent=4

The above command generates the following output:

Success.
Machine-readable output follows:
{
    "tags": [],
    "type": "vlan",
    "enabled": true,
    "system_id": "4efwb4",
    "id": 21,
    "children": [],
    "mac_address": "52:54:00:eb:f2:29",
    "params": {},
    "vlan": {
        "vid": 100,
        "mtu": 1500,
        "dhcp_on": false,
        "external_dhcp": null,
        "relay_vlan": null,
        "id": 5004,
        "secondary_rack": null,
        "fabric_id": 0,
        "space": "undefined",
        "fabric": "fabric-0",
        "name": "Storage network",
        "primary_rack": null,
        "resource_uri": "/MAAS/api/2.0/vlans/5004/"
    },
    "parents": [
        "ens3"
    ],
    "effective_mtu": 1500,
    "links": [
        {
            "id": 55,
            "mode": "link_up"
        }
    ],
    "discovered": null,
    "name": "ens3.100",
    "resource_uri": "/MAAS/api/2.0/nodes/4efwb4/interfaces/21/"
}

Delete a VLAN interface

The following command outlines the syntax required to delete a VLAN interface from the command line:

maas $PROFILE vlan delete $FABRIC__ID $VLAN_ID

Using the values from previous examples, you executed this step as follows:

maas admin vlan delete 0 100