MAAS 2.3.5 (6511-gf466fdb-0ubuntu1) - iLO 4 (moonshot issue)

joolski · 14 March 2019 13:51

Hi Maas’ers,

I’m setting up a MAAS 2.3.5 cluster with some HP machines (ProLiant DL360p Gen8) (iLo 2.61) and I’m running into a few issues getting the BMC power configuration to function correctly.

Cloud-init returns IPMI power type, rather than HP Moonshot, which means I can’t commission the device as it can’t turn it on
Using the web UI, when I set the Power configuration to HP Moonshot I get an error stating that ipmitool is not installed, I installed ipmitool and now I have a new issue.

Here’s the rackd log message :

provisioningserver.drivers.power.PowerActionError: Failed to execute (‘ipmitool’, ‘-I’, ‘lanplus’, ‘-H’, ‘10.0.0.25’, ‘-U’, ‘maas’, ‘-P’, ‘exra3TyjSmdc6u’, ‘10.0.0.25’, ‘chassis’, ‘bootdev’, ‘pxe’) for cartridge 10.0.0.25 at 10.0.0.25: Error: Unable to establish IPMI v2 / RMCP+ session
2019-03-14 13:41:51 provisioningserver.rpc.power: [critical] free-dove: Power on failed.

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 393, in callback
self._startRunCallbacks(result)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 501, in _startRunCallbacks
self._runCallbacks()
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1184, in gotResult
_inlineCallbacks(r, g, deferred)
— —
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File “/usr/lib/python3/dist-packages/twisted/python/failure.py”, line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File “/usr/lib/python3/dist-packages/provisioningserver/rpc/power.py”, line 285, in change_power_state
system_id, hostname, power_type, power_change, context)
File “/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1128, in _inlineCallbacks
result = g.send(result)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/init.py”, line 422, in perform_power
raise exc_info0.with_traceback(exc_info[2])
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/init.py”, line 379, in perform_power
power_func, system_id, context)
File “/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 246, in inContext
result = inContext.theWork()
File “/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 262, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/python/context.py”, line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File “/usr/lib/python3/dist-packages/twisted/python/context.py”, line 81, in callWithContext
return func(*args,**kw)
File “/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py”, line 232, in wrapper
result = func(*args, **kwargs)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 333, in power_on
self._issue_ipmi_command(‘on’, **context)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 330, in _issue_ipmi_command
ipmipower_command, power_change, power_address)
File “/usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py”, line 265, in _issue_ipmipower_command
raise error_info.get(‘exception’)(error_info.get(‘message’))
provisioningserver.drivers.power.PowerConnError: Connection timed out while performing power action. Check BMC configuration and connectivity and try again.
2019-03-14 13:46:51 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: enp3s0, enp2s0.
2019-03-14 13:47:02 provisioningserver.dhcp.detect: [info] External DHCP server(s) discovered on interface ‘enp3s0’: 10.0.0.1
2019-03-14 13:47:12 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.

I’ve tried to manually run the command myself and I get the same error.

If there’s more information I need to supply, just shout. I tried this exact setup about a month back and it worked a treat, however I can’t recall the versions I was using.

Any help gratefully received.

–Jools

joolski · 14 March 2019 14:00

Sorry, forgot to add the screen shot of the UI error too.

andreserl · 14 March 2019 14:21

Hi There,

To power manage machines inside HP moonshot, you have three options:

Use the IPMI power driver, which would allow MAAS to communicate with the BMC of the machines inside the chassis directly. (This is the recommended path).
You can use HP Moonshot - ILO chassis manager, which will allow you to communicate directly to the ILO chassis manager via SSH.
You can use HP Moonshot - ILO4 (IPMI), which will allow you to contact to the moonshot chassis manager directly via IPMI, and it itself would forward IPMI requests to machines inside the chassis. This, however, requires you to provide the Power Hardware Address (which would look something like - -B <cartridge_channel> -T <node_address> -b <local_chan> -t <local_address> -m 0x20 .

MAAS has internal code where it tries to determine whether a machine is running inside a moonshot chassis. It seems that in this case it was unable to detect it and it is automatically selecting IPMI, because it can detect a BMC inside the machine.

So I would recommend you do one of two things:

A. Use #2 above, whcih would allow MAAS to communicate to the moonshot chassis directly via SSH.
B. Use #1, which would allow MAAS to communicate directly to the machines (not the chassis).

B is what’s recommended. Is there any reason why MAAS cannot communicate directly to the BMC of each of the Proliant Servers inside the chassis?

joolski · 14 March 2019 14:55

Hi Andre,

Many thanks for the quick response.

I’ve tried the first option, using the IPMI driver, I used IPMI1.5 and also IPMI 2.0, but I get the error on the UI

“Error:Connection timed out while performing power action. Check BMC configuration and connectivity and try again.”

I’ve logged into the iLo console (via ssh) from both the rack and the region machine with no problems.

I’m guessing it’s the power hardware address, which is translated into a cartridge address ? I’m using the ip address of the ilo adapter, should this be something else ?

Many thanks,

–Jools

joolski · 14 March 2019 15:21

So, a little more information.

I’ve tried to use the ipmitool on the command line from both the region and the rack controller.
The results are the same. I ran the tool in debug more so you can see the packet dump, however I’ve no clue about the protocol it’s using, so hopefully it’ll make more sense to you !

I’ve not specified - -B <cartridge_channel> -T <node_address> -b <local_chan> -t <local_address> as this is a standalone server, not a chassis.

ubuntu maas-region-01:~$ ssh maas 10.0.0.24
maas 10.0.0.24’s password:

ubuntucmaas-region-01:~$ ssh maas 10.0.0.24
maas 10.0.0.24’s password:
User:maas logged-in to ILO---------z.(10.0.0.24 / FE80::9EB6:54FF:FE81:E88E)

iLO Advanced 2.61 at Jul 27 2018
Server Name: host is unnamed
Server Power: Off

</>hpiLO->

So, ssh seems fine. We have connectivity.

Here’s the command I used for ipmitool…

ipmitool -v -v -v -v -v -v -v -H 10.0.0.24 -U maas -P 8Uw3InTqA9 -I lanplus power status

And the output (sorry, there’s a fair bit)

Sending IPMI command payload
netfn : 0x06
command : 0x38
data : 0x8e 0x04