Ipmi cipher suite C17 support

art00rek · 14 September 2020 15:00

Hello all,

I would like to point out a problem that I encountered personally during the implementation. After unsuccessful enlisting several servers, I noticed that maas doesn’t support some cipher suites for ipmi connections.

By default it uses C3 while some hardware vendors consider only C17 (ipmitool’s -C17 option) as a secure protocol. I think that this problem may gradually affect even more servers because ipmi security is recently in focus for most of hardware providers.

We can workaround this by turning on the appropriate cipher suite on the host, but when provisioning a large number of machines it can be problematic and not very elegant.

Maybe it’s worth to implement this in maas.

Thanks,
Artur

ltrager · 14 September 2020 17:06

I’m actually working on adding support for IPMI encryption for MAAS 2.9. MAAS will be able to detect if an IPMI BMC key is being used, select a secure cipher to communicate with the BMC, and disable insecure ciphers.

ltrager · 19 September 2020 19:53

I just released MAAS 2.9.0 beta3 which adds support to specify the IPMI cipher suite id, K_g key, and IPMI user privilege level.

MAAS still defaults to C3 as not all machines support C17. I am working on a branch for beta4 that will allow MAAS to automatically discover the highest supported IPMI cipher suite id and K_g key automatically.

art00rek · 24 September 2020 21:10

Hi,

I’ve just switched my installation to 2.9 beta channel to test. Great work!

Thank you!

ltrager · 25 September 2020 19:48

Thanks! I just released beta4 which adds the ability to automatically detect the IPMI most secure cipher suite id the BMC supports as well as if an K_g BMC key was set. If you recommission your machine 17 should be automatically selected.

art00rek · 25 September 2020 21:40

Hi Lee,

just recommissioned one of my machines.

before recommission it was:

{
    "power_pass": "hidden",
    "power_user": "maas",
    "power_driver": "LAN_2_0",
    "power_address": "xxx.xxx.xxx.xxx",
    "power_boot_type": "efi"
}

after recommission:

{
“k_g”: “”,
“power_pass”: “hidden”,
“power_user”: “maas”,
“power_driver”: “LAN_2_0”,
“power_address”: “xxx.xxx.xxx.xxx”,
“cipher_suite_id”: “3”,
“power_boot_type”: “efi”,
“privilege_level”: “ADMIN”
}

and at this moment I lost control of the host…

Had to manually change:

maas $PROFILE machine update $SystemID power_parameters_cipher_suite_id=“17”

to regain it back.

I also noticed that I still have C3 enabled on the channel:

RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max : XXXaXXXXXXaXXXX

ltrager · 26 September 2020 00:11

In MAAS 2.9 I moved BMC configuration to a new commissioning script, 30-maas-01-bmc-config. This script lists the available IPMI cipher suites ids, enables the most secure cipher suite, and disables any insecure cipher suite. From that output it looks like 17 isn’t being detected so 3 is being enabled and used.

Could you post the following:

The output of the commissioning script 30-maas-01-bmc-config
The output of the command bmc-config --checkout -S Rmcpplus_Conf_Privilege

art00rek · 26 September 2020 09:06

It seems that C17 is not detected… below requested outputs:

30-maas-01-bmc-config

INFO: Loading IPMI kernel modules...
INFO: Checking for HP Moonshot...
INFO: Checking for IPMI...
INFO: IPMI detected!
INFO: Verifying BMC is accessible over the network...
INFO: Configuring IPMI cipher suite ids...
INFO: MAAS will use IPMI cipher suite id "3" for BMC communication
WARNING: No K_g BMC key found or configured, communication with BMC will not use a session key!
INFO: Found existing IPMI user "maas"!
INFO: Configuring IPMI BMC user "maas"...
INFO: IPMI user number - User2
INFO: IPMI user privilege level - Administrator
INFO: IPMI Version - LAN_2_0
INFO: IPMI boot type - efi

ubuntu@alive-rat:~$ sudo  bmc-config --checkout -S Rmcpplus_Conf_Privilege
#
# Section Rmcpplus_Conf_Privilege Comments 
#
# If your system supports IPMI 2.0 and Serial-over-LAN (SOL),cipher suite IDs 
# may be configurable below. In the Rmcpplus_Conf_Privilege section, maximum 
# user privilege levels allowed for authentication under IPMI 2.0 (including 
# Serial-over-LAN) are set for each supported cipher suite ID. Each cipher suite 
# ID supports different sets of authentication, integrity, and encryption 
# algorithms for IPMI 2.0. Typically, the highest privilege level any username 
# configured should set for support under a cipher suite ID. This is typically 
# "Administrator". 
#
Section Rmcpplus_Conf_Privilege
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_1           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_2           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_3           Administrator
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_6           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_7           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_8           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_11          Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_12          Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_15          Unused
EndSection

C17 and C3 were enabled on host - but maas was not able to detect/change power state

ipmitool -H 10.77.0.171 -U maas -P <SECRET> -I lanplus -C17 lan print 1 | grep Cipher
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : XXXaXXXXXXaXXXX

I have disabled C3 leaving C17 as only cipher set on host.

ipmitool -H xxx.xxx.xxx.xxx -U maas -P <SECRET> -I lanplus -C17 lan set 1 cipher_privs XXXXXXXXXXaXXXX

Manually changed power configuration and recommissioned machine.

after that

30-maas-01-bmc-config

INFO: Loading IPMI kernel modules...
INFO: Checking for HP Moonshot...
INFO: Checking for IPMI...
INFO: IPMI detected!
INFO: Verifying BMC is accessible over the network...
INFO: Configuring IPMI cipher suite ids...
INFO: MAAS will use IPMI cipher suite id "3" for BMC communication
WARNING: No K_g BMC key found or configured, communication with BMC will not use a session key!
INFO: Found existing IPMI user "maas"!
INFO: Configuring IPMI BMC user "maas"...
INFO: IPMI user number - User2
INFO: IPMI user privilege level - Administrator
INFO: IPMI Version - LAN_2_0
INFO: IPMI boot type - efi

and

ubuntu@alive-rat:~$ sudo  bmc-config --checkout -S Rmcpplus_Conf_Privilege
#
# Section Rmcpplus_Conf_Privilege Comments 
#
# If your system supports IPMI 2.0 and Serial-over-LAN (SOL),cipher suite IDs 
# may be configurable below. In the Rmcpplus_Conf_Privilege section, maximum 
# user privilege levels allowed for authentication under IPMI 2.0 (including 
# Serial-over-LAN) are set for each supported cipher suite ID. Each cipher suite 
# ID supports different sets of authentication, integrity, and encryption 
# algorithms for IPMI 2.0. Typically, the highest privilege level any username 
# configured should set for support under a cipher suite ID. This is typically 
# "Administrator". 
#
Section Rmcpplus_Conf_Privilege
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_1           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_2           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_3           Administrator
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_6           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_7           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_8           Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_11          Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_12          Unused
	## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
	Maximum_Privilege_Cipher_Suite_Id_15          Unused
EndSection

and again maas is not able to control host power, but this time c3 was not enabled.

ipmitool -H xxx.xxx.xxx.xxx -U maas -P <SECRET> -I lanplus -C17 lan print 1 | grep Cipher
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : XXXXXXXXXXaXXXX
                        :     X=Cipher Suite Unused

ltrager · 28 September 2020 23:38

Interesting Rmcpplus_Conf_Privilege only shows ciphers 1-15 while ipmitool shows 1-17. On my test hosts those two are identical. I went with Rmcpplus_Conf_Privilege since its a bit easier to understand. I’ll look into reworking the code.

Thanks for sending me the output!

art00rek · 29 September 2020 10:23

Am I right that you use freeipmi? I’m not sure that it supports anything above C15…

ltrager · 29 September 2020 21:48

Yes MAAS uses freeipmi-tools. According to the freeipmi-tools changelog support for IPMI cipher suites 15, 16, and 17 was added in 1.2.1 which was released on 08/22/12. Ubuntu Xenial 16.04 LTS and Ubuntu Bionic 18.04 LTS both have freeipmi-tools 1.4.11 while Ubuntu Focal 20.04 LTS has freeipmi-tools 1.6.4.

BMCs can be a bit inconsistent with their output and support which may be why cipher suite 17 shows up with ipmitool and not bmc-config.

ltrager · 16 October 2020 20:31

I added a patch to MAAS 2.9.0 beta 6 which should fix this issue. Both bmc-config and ipmitool are now used to detect the IPMI cipher id. Whichever has the highest id is used.

Let me know if that fixes things for you!

rbur004 · 26 February 2021 02:26

This is breaking my IBM dx360 M4 and X3650 M4 ipmi nodes. If I commission them, the IPMI cipher flags change, and MAAS can no longer connect. It also breaks the asu64 command.

Looking at a host’s ipmi, before MAAS commissions it (with ipmitool lan print), it has
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : XaaaaaaaaaaaaaX

After MAAS commissions it, it has
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : XXXaXXXXaXXXaXX

I had to use ‘lan set 1 cipher_privs XaaaaaaaaaaaaaX’ to allow MAAS to connect to the node again.

Looks like a 1 off error, as XXaXXXXaXXXaXXX works too

ltrager · 26 February 2021 17:09

We’re tracking an issue with Lenovo BMC’s in LP:1916860.

rbur004 · 28 February 2021 19:51

Good to see it’s been reported. I get an error logging in to the bug tracking site, so can’t add my 2c worth

The code probably needs to check the existing cipher suites list, before changing the mask. We have several generations of IBM (Lenovo) servers, and the mask’s X’s look to be lined up with the reported cyphers list.

That might be different on another manufacturer’s servers though, so the quick fix would be to not touch the existing mask. At least until any variations in nodes IPMI’s is understood.

The real security for the IPMI interface, is to ensure they are on a private network. Network access should be restricted to just the provisioning service.

jhusakowski · 28 June 2023 08:27

MAAS 2.9 and later try to find the most secure cipher suite available. Preference order is 17, 3, 8, 12. If detection fails MAAS will fall back to using freeipmi-tools default, 3, which is what previous versions of MAAS use. The users can also specify cipher 17 as a default.

system · 30 June 2023 08:27

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.