MAAS 2.3.5 (6511-gf466fdb-0ubuntu1) - iLO 4 (moonshot issue)


#1

Hi Maas’ers,

I’m setting up a MAAS 2.3.5 cluster with some HP machines (ProLiant DL360p Gen8) (iLo 2.61) and I’m running into a few issues getting the BMC power configuration to function correctly.

  1. Cloud-init returns IPMI power type, rather than HP Moonshot, which means I can’t commission the device as it can’t turn it on :frowning_face:

  2. Using the web UI, when I set the Power configuration to HP Moonshot I get an error stating that ipmitool is not installed, I installed ipmitool and now I have a new issue.

Here’s the rackd log message :

provisioningserver.drivers.power.PowerActionError: Failed to execute (‘ipmitool’, ‘-I’, ‘lanplus’, ‘-H’, ‘10.0.0.25’, ‘-U’, ‘maas’, ‘-P’, ‘exra3TyjSmdc6u’, ‘10.0.0.25’, ‘chassis’, ‘bootdev’, ‘pxe’) for cartridge 10.0.0.25 at 10.0.0.25: Error: Unable to establish IPMI v2 / RMCP+ session
2019-03-14 13:41:51 provisioningserver.rpc.power: [critical] free-dove: Power on failed.

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 393, in callback
self._startRunCallbacks(result)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 501, in _startRunCallbacks
self._runCallbacks()
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1184, in gotResult
_inlineCallbacks(r, g, deferred)
— —
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File “/usr/lib/python3/dist-packages/twisted/python/failure.py”, line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File “/usr/lib/python3/dist-packages/provisioningserver/rpc/power.py”, line 285, in change_power_state
system_id, hostname, power_type, power_change, context)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1128, in _inlineCallbacks
result = g.send(result)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/init.py”, line 422, in perform_power
raise exc_info0.with_traceback(exc_info[2])
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/init.py”, line 379, in perform_power
power_func, system_id, context)
File “/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 246, in inContext
result = inContext.theWork()
File “/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 262, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/python/context.py”, line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/python/context.py”, line 81, in callWithContext
return func(*args,**kw)
File “/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py”, line 232, in wrapper
result = func(*args, **kwargs)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 333, in power_on
self._issue_ipmi_command(‘on’, **context)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 330, in _issue_ipmi_command
ipmipower_command, power_change, power_address)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 265, in _issue_ipmipower_command
raise error_info.get(‘exception’)(error_info.get(‘message’))
provisioningserver.drivers.power.PowerConnError: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-03-14 13:46:51 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: enp3s0, enp2s0.
2019-03-14 13:47:02 provisioningserver.dhcp.detect: [info] External DHCP server(s) discovered on interface ‘enp3s0’: 10.0.0.1
2019-03-14 13:47:12 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.

I’ve tried to manually run the command myself and I get the same error.

If there’s more information I need to supply, just shout. I tried this exact setup about a month back and it worked a treat, however I can’t recall the versions I was using.

Any help gratefully received.

–Jools


#2

Sorry, forgot to add the screen shot of the UI error too.


#3

Hi There,

To power manage machines inside HP moonshot, you have three options:

  1. Use the IPMI power driver, which would allow MAAS to communicate with the BMC of the machines inside the chassis directly. (This is the recommended path).
  2. You can use HP Moonshot - ILO chassis manager, which will allow you to communicate directly to the ILO chassis manager via SSH.
  3. You can use HP Moonshot - ILO4 (IPMI), which will allow you to contact to the moonshot chassis manager directly via IPMI, and it itself would forward IPMI requests to machines inside the chassis. This, however, requires you to provide the Power Hardware Address (which would look something like - -B <cartridge_channel> -T <node_address> -b <local_chan> -t <local_address> -m 0x20 .

MAAS has internal code where it tries to determine whether a machine is running inside a moonshot chassis. It seems that in this case it was unable to detect it and it is automatically selecting IPMI, because it can detect a BMC inside the machine.

So I would recommend you do one of two things:

A. Use #2 above, whcih would allow MAAS to communicate to the moonshot chassis directly via SSH.
B. Use #1, which would allow MAAS to communicate directly to the machines (not the chassis).

B is what’s recommended. Is there any reason why MAAS cannot communicate directly to the BMC of each of the Proliant Servers inside the chassis?


#4

Hi Andre,

Many thanks for the quick response.

I’ve tried the first option, using the IPMI driver, I used IPMI1.5 and also IPMI 2.0, but I get the error on the UI

“Error:Connection timed out while performing power action. Check BMC configuration and connectivity and try again.”

I’ve logged into the iLo console (via ssh) from both the rack and the region machine with no problems.

I’m guessing it’s the power hardware address, which is translated into a cartridge address ? I’m using the ip address of the ilo adapter, should this be something else ?

Many thanks,

–Jools


#5

So, a little more information.

I’ve tried to use the ipmitool on the command line from both the region and the rack controller.
The results are the same. I ran the tool in debug more so you can see the packet dump, however I’ve no clue about the protocol it’s using, so hopefully it’ll make more sense to you !

I’ve not specified - -B <cartridge_channel> -T <node_address> -b <local_chan> -t <local_address> as this is a standalone server, not a chassis.

ubuntu maas-region-01:~$ ssh maas 10.0.0.24
maas 10.0.0.24’s password:

ubuntucmaas-region-01:~$ ssh maas 10.0.0.24
maas 10.0.0.24’s password:
User:maas logged-in to ILO---------z.(10.0.0.24 / FE80::9EB6:54FF:FE81:E88E)

iLO Advanced 2.61 at Jul 27 2018
Server Name: host is unnamed
Server Power: Off

</>hpiLO->

So, ssh seems fine. We have connectivity.

Here’s the command I used for ipmitool…

ipmitool -v -v -v -v -v -v -v -H 10.0.0.24 -U maas -P 8Uw3InTqA9 -I lanplus power status

And the output (sorry, there’s a fair bit)

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x8e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 8e 04 b5
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x8e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 8e 04 b5
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x8e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 8e 04 b5
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x8e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 8e 04 b5
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x0e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 0e 04 35
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x0e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 0e 04 35
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x0e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 0e 04 35
removed list entry seq=0x00 cmd=0x38

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x0e 0x04

BUILDING A v1.5 COMMAND
added list entry seq=0x00 cmd=0x38

IPMI Request Session Header
Authtype : NONE
Sequence : 0x00000000
Session ID : 0x00000000
IPMI Request Message Header
Rs Addr : 20
NetFn : 06
Rs LUN : 0
Rq Addr : 81
Rq Seq : 00
Rq Lun : 0
Command : 38
sending packet (23 bytes)
06 00 ff 07 00 00 00 00 00 00 00 00 00 09 20 18
c8 81 00 38 0e 04 35
removed list entry seq=0x00 cmd=0x38
Get Auth Capabilities error
Error issuing Get Channel Authentication Capabilities request
Error: Unable to establish IPMI v2 / RMCP+ session


#6

So a moonshoot chassis will have the IP for the chassis itself, and each BMC of each of the machines will have its own IP address. My understanding is that depending on how it is configured, the IP of the machines BMCs would be “internal” to the chassis or external to the network.

If they are external to the network, I would use option B above.

If they are internal to the chassis, I would use option A, or if you can figure out the cartridge address and all other details (e.g. - -B <cartridge_channel> -T <node_address> -b <local_chan> -t <local_address> -m 0x20)


#7

Hi Andre,

Thanks for the feedback, unfortunately I’m a bit stuck now. Looks like the ipmitool version 1.8.16 does not seem to work as per the documentation.

I’m going to see if I can build a previous version of the tool and try that, and post up the results.

Thanks for all your help, much appreciated.

–J


#8

So… Seems like I missed an option on the iLO 4 setup to enable IPMI/DCMI over lan access.

As soon as I enabled this option ipmitool worked perfectly.

It might be useful to check if this option is enabled when commissioning the machine as it’s quite confusing trying to understand what the issue is.