MAAS 2.6.0 released!


#1

MAAS 2.6.0 (final)

We are happy to announce that MAAS 2.6.0 is now available and introduces quite a few interesting features as well as several improvements. See below!

Important announcements

Deploying machines with multiple “default” gateways

Deploying machines in complex network scenarios, where you want traffic to go out through a different gateway other than the default gateway, requires machines to be configured with source routing. This ensures that any outgoing traffic for which there are no defined routes goes out through the same interface as the incoming traffic.

MAAS 2.6 now changes the default machine network configuration to take advantage of source routing.

PXE booting leveraging HTTP

MAAS 2.6 changes the default underlying mechanism by which machines obtain files over the PXE boot process. For amd64 legacy mode, MAAS will now only download lpxelinux.0 over TFTP and the rest of the files and config via HTTP. MAAS now supports the 2.5 UEFI spec which provides full HTTP support over grub. arm64 is expected to work the same way as amd64, and ppc64el already uses HTTP to download kernel and initrd.

The major change, however, is for Virtual Machines on amd64, in which we know rely entirely on iPXE to do HTTP boot.

New features & improvements

Multiple default gateways

Previous MAAS versions deployed machines with a single default gateway. This meant that if a machine was configured in multiple subnets with gateways (with or without any routes), all incoming traffic on a given gateway would go out through the default gateway instead of the intended gateway.

From MAAS 2.6, this behavior now changes as MAAS now leverages the use of netplan and source routing to configure multiple “default” gateways, ensuring that all outgoing traffic goes out through the correct gateway rather than always going out through the default gateway.

For example, in previous versions of MAAS if a machine was with two subnets (subnet1, subnet2), and using subnet1’s gateway as the default gateway, all traffic in subnet two to the outside world would go out through the default gateway. Instead, with the newer changes, all traffic in subnet2 will go out through subnet2’s gateway in case it needs to reach the outside world.

Improving the MAAS PXE boot process with HTTP

MAAS has always relied on TFTP to support the PXE boot process. Starting from 2.5 we started transitioning some of the boot process to use HTTP rather than full TFTP. Doing so significantly increases the boot speed and performance, provided that TFTP is a slow protocol in comparison to HTTP.

KVMs will now use iPXE and HTTP (instead of pxelinux/tftp) for AMD64

Previously MAAS deployed AMD64 KVMs using pxelinux emulation. This meant that the VM would be told by the DHCP server to get lpxelinux.0 and other files from TFTP. Starting from 2.6, however, KVM deployment now relies on iPXE and HTTP instead. This means that:

  • MAAS will autoconfigure the DHCP to tell VM’s using iPXE to boot off a HTTP endpoint.
  • iPXE on the VM will obtain the boot configuration file over HTTP.
  • iPXE will obtain linux/kernel over HTTP (please note that as of 2.5, VMs were already obtaining this over HTTP leveraging the use of lpxelinux.0 instead, but still relied on TFTP to obtain all pxelinux related roms).

Legacy boot for AMD64 now mostly relies on HTTP

In MAAS 2.5, physical systems would rely on TFTP to download lpxelinux.0, all required roms and the configuration. Starting from 2.6, machines will now only rely on TFTP to download lpxelinux.0, after which all the remaining roms and configuration files will be obtained from an HTTP endpoint.

What about EFI?

MAAS 2.6 now supports the UEFI 2.5 spec which provides full HTTP boot.

What about other arches?

As of MAAS 2.6, ppc64el has supported HTTP boot to download the kernel/initird. The configuration file continues to be obtained over TFTP.

As far as arm64, given that it uses grub, we leverage the same HTTP boot support.

Expanding ESXi support

ESXi Storage configuration

In 2.5, MAAS introduced the ability to deploy ESXi. This involved only having the ability to deploy on the root disk, and didn’t provide the ability to use the others from MAAS. Starting from 2.6, however, MAAS is now able to configure advanced storage for ESXi deployments. This means that admins are able to select the root, and create datastores (for VM storage) directly from MAAS. This feature, however, does require a newer version of the image generation scripts.

Please note that this feature is currently only available over the API. The UI will come at a later cut. For more information about ESXi in MAAS, please contact Canonical at https://docs.maas.io/devel/en/installconfig-images-vmware .

ESXi vCenter registration

MAAS 2.6b1 now adds the ability for ESXi deployments to automatically register themselves into a vCenter. This is done in several ways, that is by baking the credentials into the images, or by providing the credentials to the machine via MAAS.

With the latter approach, MAAS now has the ability to store vCenter credentials (which should be scoped down to only have permissions to register new ESXi hosts), which are provided to the machine during the deployment process. The hosts will then use these credentials to register themselves.

For more information about ESXi in MAAS, please refer to https://docs.maas.io/devel/en/installconfig-images-vmware.

Minor features and improvements

Better boot process output

MAAS 2.6b1 introduces a better boot process output. MAAS has cues in which it allows users to better understand the machine deployment process. In MAAS 2.6b1, we are improving the process output which will yield a better insight of where machines are throughout the deployment and releasing process.

Further work will provide better insight for other states, such as commissioning and hardware testing.

Ephemeral deployment for disk-less machines

In older versions, MAAS prevented machines without any attached disks to be deployed. As of 2.6, MAAS now allows deployment all diskless machines to be ephemerally deployed, by default, which simply runs the operating system in memory. This feature provides the ability to:

  • Network interface configuration - MAAS will apply the network interface configuration on the ephemeral environment.
  • Deployment customization via cloud-init (user-data) - MAAS will continue to allow users to provide user-data to customize their deployments.
  • Reconfiguration on reboot - On reboot, an ephemerally deployed machined will re-run all configuration that was provided to cloud-init, including that provided by the user (user-data).

Please note that ephemeral deployments are limited in that:

  • Ubuntu as the only ephemeral OS.
  • Curtin preseeds are no longer applicable for deployment configuration.
  • Storage is no longer configured.

PXE over UUID

MAAS now supports legacy PXE booting over UUID. The MAAS commissioning process will now gather the machine’s UUID, which is also used for the PXE process over pxelinux. MAAS, by default, will now prefer to respond to PXE requests based on the UUID rather than the MAC address. This was originally added to start supporting s390x deployment.

Ability to suppress failed test result (API only)

When a hardware tests fails, MAAS provides visual cues that highlight the machine to ensure that admins know that there’s been a problem with a specific machine. However, in some situations, admins have noted that they would like a mechanism to suppress the error notice of the given test. As such, this release introduces the ability to suppress this error over the API only.
In a subsequent release of MAAS 2.6, this will be added to the UI.

Add description field to the node model (API only)

MAAS adds a new admin-editable field to the node model in MAAS. This field is to allow admins to write notes or provide descriptions for a machine. This is informational only and
only supported over the API.
In the subsequent release of MAAS 2.6 this will be added to the UI.

Expanding the Ecosystem

Redfish power management

MAAS now adds the ability to power manage machines via Redfish, and has been tested against HP and Dell servers. Please note that most BMC’s continue to support IPMI alongside Redfish REST APIs for remote management of the BMC, as such a few considerations:

  • If the machine is automatically enlisted, MAAS will default to discover/configure IPMI.
  • If a machine has been switched to use Redfish, or created with the Redfish power driver, MAAS will always use this and won’t automatically switch back to IPMI.
  • It allows defining a node ID in case the machine is inside a chassis management interface with multiple machines.

OpenBMC power driver

MAAS 2.6 has added a new power driver. This is the OpenBMC REST power driver, which allows MAAS to power manage machines via the REST interface.

Minor Performance improvements

As a result of a bugfix where MAAS was creating database records for interface links against null IP address, the UI performance has been increased. This underlying issue was causing load times of network elements in the UI. The fix has reduced the queries from 30s to 1s. This improves the perceived performance in large environments where there’s significant DHCP traffic.

Stats and metrics gathering

MAAS is adding support to collect metrics and stats about the environment and performance. This will all users to have more insight about usage and performance, as well as help developers use this data to further improve performance.

Stats

The following are the available stats:

  • Machine counts by status
  • Machine counts by type
  • Machine counts by architecture
  • Counts of network elements
  • spaces
  • fabrics
  • VLANs
  • IPv4 subnets
  • IPv6 subnets
  • Combined memory size for all machines
  • Combined number of CPUs for all machines
  • Combined storage size all machines
  • Number of KVM pods
  • Number of KVM virtual machines
  • Number of KVM cores
  • Memory for KVM pods
  • Storage size for KVM pods

Performance metrics

The following are the available performance metrics:

  • CPU usage counters
  • Memory usage counters
  • HTTP requests latency
  • HTTP requests response size
  • HTTP requests query count
  • HTTP requests query latency
  • Websocket calls latency
  • RPC calls latency
  • TFTP files transfer latency

Web UI

Machine listing group by

The machine listing page now has the ability to group machines in a number of different ways:

  • Status - Machines will be grouped according to their status
  • User - Machines will be grouped by the owner
  • No grouping - allows the same older view.

In-table actions on machine listing

The machine listing page includes in-table dropdown actions list for quick operations without having to enter the details on the machine. The actions available are; execute power actions, apply available actions, change pool, zone, and owner on the machine. These can be found on the cell of each row they apply too.

Bulk tagging

MAAS 2.6 introduces the ability to bulk tag machines via the Web UI. Simply select the machines in the machine listing and hit tag in the actions menu.

Add a note (description) to machine

It is now possible for administrators to add a note to the machine. Adding a note is currently exposed under the ‘Configuration’ tab of the machine details page. The note is visible on the machine listing page and can be searched for using the search bar on the page.

Display space in machine listing

If a machine is in a space, this is information is now visible in the machine listing table.

ESXi storage layout and custom storage layout per machine

Following up on the ESXi support provided by the API, users can now set the machine storage layout to VMFS6 in the storage tab for each individual machine from the UI. MAAS then creates all the necessary partitions and a datastore in order for an ESXi image to boot. The user is then able to create additional datastores or add more disks to the newly created one.

Also, users are able to now change the storage layout applied to each machine through the Web UI.

Redesign DHCP management

2.6a2 introduces a better design to enable, modify, and disable DHCP over the Web UI and improves the overall user experience moving the process in-section rather than as an action in the VLAN details page.

Suppress failed tests

Sometimes hardware test on a machine might be displaying a result that’s no longer valid or show a failure that’s not true anymore. In order to remove unnecessary warnings in the machine listing page, users can choose to suppress the current failures. Hiding them from the machine listing and machine summary. All the results are still visible in the hardware testing tab of the machine and can be un-suppressed via the checkboxes next to each test.

Add search and filters to the dashboard

This release introduces a new search and filter to the dashboard view where all discovered devices are available.

Clear discovered devices

The device discovery page now has the ability to clear all discovered records, or clear individual ones.

Form layout improvements

The layout of forms (both editable and non-editable) has been visually improved to allow users to focus on the data.

Installation

MAAS 2.6.0 is available for Ubuntu Cosmic, Bionic & Disco and can be installed from our PPA as follows:

sudo apt-add-repository ppa:maas/stable
sudo apt install maas

**NOTE**: Please note MAAS 2.6.0 has now replaced MAAS 2.5 in ppa:maas/stable.


Future feature: Dashboard
What's new in 2.6
PXE not working after upgrading to 2.5.0
#2

Awsome work team i am testing the esxi flow as we speak.

I am wondering if i am able to provide options to provide a vmotion vmk information are you able to generate that in the KS.cfg

i can provide examples and willing to help.


#3

Looking forward to looking at the prometheus metrics!

I think a couple of things broke on xenial, or atleast I’m having some issues. Create a bridge on your primary pxe interface, and assign your primary ip to it. When deploying xenial the bridge-utils aren’t installed, so when it reboots after install the interface won’t come up and “Can not apply stage config, no datasource found! Likely bad things to come!”

Right now I’m working around it by adding a late_command in-target apt install bridge-utils to the /etc/maas/preseed/curtin_userdata, but maybe it should go in maasserver/preseed.py as an exception/special case for xenial?