Best way to load kernel module during enlist/commission

Users
1

Hi all

I want to see if I can trick MAAS into showing me our Infiniband Cards.
For that, I want to try to have the ephemeral boot environment load the ib_ipoib kernel module and leave it to MAAS to see if it finds something. (I mean, it also finds stuff like iDRAC usb modules, so…)

What’s my best starting point here?
I am pondering any of the following options

  • Using the enlist preseed to write an /etc/modprobe.d/ file (this might be too late?)
  • Using the enlist preseed to eventually call modprobe ib_ipoib || true (this feels wrong)
  • Trying to figure out a kernel command line that somehow effects the loading of the module (i haven’t found one yet)

I’m asking first because (a) rebooting any of our IB-capable hosts takes a lot of time and (b) maybe someone else can benefit from this.

-Tobias

2

bump

Someone any idea?

3

hi @score since commissioning image is ephemeral if found that i need to install the following before to have some other module loaded

apt-get -y install “linux-modules-$(uname -r)”

for ib_ipoib you will need also install the following

apt-get -y install rdma-core

4

Hi

yes, that makes sense!
where would you put that?

5 
#!/bin/bash
# --- Start MAAS 1.0 script metadata ---
# name: 01-test-02-ib
# title: test
# description: Test try
# script_type: commissioning
# recommission: False
# destructive: False
# hardware_type: node
# may_reboot: False
# parallel: disabled
# --- End MAAS 1.0 script metadata ---
set -eux
set -o pipefail

apt-get -y install "linux-modules-$(uname -r)"
apt-get -y install rdma-core
lsmod | grep -E "ib|mlx|rdma"

this expose some extra info in machine xml

without commissioning script

        <lshw:node id="network:0" claimed="true" class="network" handle="PCI:0000:04:00.0" modalias="pci:v000015B3d00001021sv000015B3sd00000127bc02sc07i00">
         <lshw:description>Infiniband controller</lshw:description>
         <lshw:product>MT2910 Family [ConnectX-7]</lshw:product>
         <lshw:vendor>Mellanox Technologies</lshw:vendor>
         <lshw:physid>0</lshw:physid>
         <lshw:subproduct>Mellanox Technologies</lshw:subproduct>
         <lshw:subvendor>Mellanox Technologies</lshw:subvendor>
         <lshw:businfo>pci@0000:04:00.0</lshw:businfo>
         <lshw:version>00</lshw:version>
         <lshw:width units="bits">64</lshw:width>
         <lshw:clock units="Hz">33000000</lshw:clock>
         <lshw:configuration>
          <lshw:setting id="driver" value="mlx5_core"/>
          <lshw:setting id="latency" value="0"/>
         </lshw:configuration>
         <lshw:capabilities>
          <lshw:capability id="pciexpress">PCI Express</lshw:capability>
          <lshw:capability id="vpd">Vital Product Data</lshw:capability>
          <lshw:capability id="msix">MSI-X</lshw:capability>
          <lshw:capability id="pm">Power Management</lshw:capability>
          <lshw:capability id="bus_master">bus mastering</lshw:capability>
          <lshw:capability id="cap_list">PCI capabilities listing</lshw:capability>
          <lshw:capability id="rom">extension ROM</lshw:capability>
         </lshw:capabilities>
         <lshw:resources>
          <lshw:resource type="irq" value="81"/>
          <lshw:resource type="memory" value="92000000-93ffffff"/>
          <lshw:resource type="memory" value="99f00000-99ffffff"/>
          <lshw:resource type="memory" value="96000000-97ffffff"/>
         </lshw:resources>
         <lshw:hints>
          <lshw:hint name="icon" value="network"/>
          <lshw:hint name="pci.class" value="0x207"/>
          <lshw:hint name="pci.device" value="0x1021"/>
          <lshw:hint name="pci.subdevice" value="0x127"/>
          <lshw:hint name="pci.subvendor" value="0x15B3"/>
          <lshw:hint name="pci.vendor" value="0x15B3"/>
         </lshw:hints>
        </lshw:node>

with commissioning script

        <lshw:node id="network:0" claimed="true" class="network" handle="PCI:0000:04:00.0" modalias="pci:v000015B3d00001021sv000015B3sd00000127bc02sc07i00">
         <lshw:description>interface</lshw:description>
         <lshw:product>MT2910 Family [ConnectX-7]</lshw:product>
         <lshw:vendor>Mellanox Technologies</lshw:vendor>
         <lshw:physid>0</lshw:physid>
         <lshw:subproduct>Mellanox Technologies</lshw:subproduct>
         <lshw:subvendor>Mellanox Technologies</lshw:subvendor>
         <lshw:businfo>pci@0000:04:00.0</lshw:businfo>
         <lshw:logicalname>ibp4s0f0</lshw:logicalname>
         <lshw:version>00</lshw:version>
         <lshw:serial>00:00:10:69:fe:80:00:00:00:00:00:00:a0:88:00:00:00:00:00:00</lshw:serial>
         <lshw:width units="bits">64</lshw:width>
         <lshw:clock units="Hz">33000000</lshw:clock>
         <lshw:configuration>
          <lshw:setting id="autonegotiation" value="off"/>
          <lshw:setting id="broadcast" value="yes"/>
          <lshw:setting id="driver" value="mlx5_core[ib_ipoib]"/>
          <lshw:setting id="driverversion" value="5.15.0-117-generic"/>
          <lshw:setting id="duplex" value="full"/>
          <lshw:setting id="firmware" value="28.42.1000 (MT_0000001045)"/>
          <lshw:setting id="latency" value="0"/>
          <lshw:setting id="link" value="yes"/>
          <lshw:setting id="multicast" value="yes"/>
         </lshw:configuration>
         <lshw:capabilities>
          <lshw:capability id="pciexpress">PCI Express</lshw:capability>
          <lshw:capability id="vpd">Vital Product Data</lshw:capability>
          <lshw:capability id="msix">MSI-X</lshw:capability>
          <lshw:capability id="pm">Power Management</lshw:capability>
          <lshw:capability id="bus_master">bus mastering</lshw:capability>
          <lshw:capability id="cap_list">PCI capabilities listing</lshw:capability>
          <lshw:capability id="rom">extension ROM</lshw:capability>
          <lshw:capability id="physical">Physical interface</lshw:capability>
         </lshw:capabilities>
         <lshw:resources>
          <lshw:resource type="irq" value="79"/>
          <lshw:resource type="memory" value="92000000-93ffffff"/>
          <lshw:resource type="memory" value="99f00000-99ffffff"/>
          <lshw:resource type="memory" value="96000000-97ffffff"/>
         </lshw:resources>
         <lshw:hints>
          <lshw:hint name="icon" value="network"/>
          <lshw:hint name="pci.class" value="0x207"/>
          <lshw:hint name="pci.device" value="0x1021"/>
          <lshw:hint name="pci.subdevice" value="0x127"/>
          <lshw:hint name="pci.subvendor" value="0x15B3"/>
          <lshw:hint name="pci.vendor" value="0x15B3"/>
         </lshw:hints>
        </lshw:node>

you can set an automated tag and call it “MLNX_IB_2P”
Looking for machines with a Mellanox vendor id and device class InfiniBand which also has 2 ports

count(//lshw:node[lshw:hints/lshw:hint[@name='pci.class' and @value='0x207'] and lshw:hints/lshw:hint[@name='pci.vendor' and @value='0x15B3']]) = 2

A more generic MLNX_IB tag

//lshw:node[lshw:hints/lshw:hint[@name='pci.class' and @value='0x207'] and lshw:hints/lshw:hint[@name='pci.vendor' and @value='0x15B3']]
1 Like
6

Wow, thats great!
Thanks a lot. I’m marking as solved (tho, I had not time to test it yet)

1 Like
7

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.