I’m trying to set up power management through Intel AMT on MAAS, and i’m hitting a wall. The check power, power on, power off commands do not work, but I’m sure AMT is enabled and functioning as using MeshCentral to perform such tasks just work.
I’m selecting :
Power type : Intel AMT
Power password : The Password for the AMT (the username is admin)
Power address : 10.0.0.190 (the address is obtained through the maas DHCP)
How to debug this ?
For information, MeshCentral tells me the version of the Intel ME are these one : v16.1.25, TLS
I’m on MAAS 3.4.0~rc1.
Also, a bit of log from maas :
2023-07-17 15:36:49 maasserver.websockets.handlers.machine: [critical] Failed to update power state of machine.
Traceback (most recent call last):
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 1174, in eb_unknown
failure.trap(UnknownPowerType, NotImplementedError)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
error.trap(RemoteAmpError)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
twisted.internet.defer.CancelledError:
Note : it seems meshcommander only works with the --tls flag, e.g. # ./meshcmd AmtFeatures --password Password1234! --host 10.0.0.190 --user admin --tls works but not # ./meshcmd AmtFeatures --password Password1234! --host 10.0.0.190 --user admin --tls , so maybe that’s related ?
I will check if we have machine with Intel AMT in order to give it a try.
Power management is done by Rack Controller and meanwhile I can recommend you to check /var/snap/maas/common/log/rackd.log if you have MAAS installed as a snap, or /var/log/maas/rackd.log if you have a deb installation.
You can inspect AMT power driver code here, under the hood MAAS is using amttool
Can you try using amttool directly, just to check that it actually works. It might be that version bundled with MAAS doesn’t work for your case.
Note: Starting with AMT 9.0 Intel has removed support for the old SOAP protocol which is
used by amttool. Only WS Management (added in AMT 3.0) is supported by recent machines.
You can use wsmancli (see http://openwsman.sf.net, your distro should have packages ready)
instead.
And indeed, using directly amttool doesn’t give any result. But it seems there is some support of wsman in the packages too. I’ll try to debug it and report back.
Right, for AMT >= 9.0 it should use wsman according to this logic. Maybe _get_wsman_command should pass --tls as well? Or maybe something should be tweaked in the openwsman_client.conf
I was in the same boat a while ago and found out that there are a couple of things that needs to be changed on the MAAS host in order to fix this little problem with AMT.
So firstly the new Intel AMT devices have enhanced security and force you to use the TLS port (16993). Although you don’t need a valid certificate, it just means that communicating to them via WSman becomes more tedious.
So in order to allow this communication you need to install wsman on the Maas host (sudo apt install wsmancli)
Secondly, you will need to make sure that your host will allow curl to do UnsafeLegacy connections. To do this, edit /etc/ssl/openssl.cnf and at the end, add the following lines:
This will allow wsman to talk to Intel AMT over 16993 using TLS.
Lastly you need to update the amt power driver (I think maas should probably allow this by default) to swap between both http and https: https://pastebin.com/zH8KNS3K
This was added to /usr/lib/python3/dist-packages/provisioningserver/drivers/power/amt.py and it only works for dep installs, if you use snap the changes will need to exist in the maas package, so for now it only works on deb
Go over the script before using it, but essentially it will add a new parameter in the Intel AMT power module called schema and you can select http or https. For the new devices, using https is the only way it will communicate.
Lastly restart maas to ensure the new driver gets loaded. That should set you to run with Intel AMT as a power manager
In the future, I want to create an module that will move away from wsman. I think it might need to be a new driver altogether as I don’t know if all devices will support it.
I have explored this protocol and think it might be nice to use requests rather than wsman-cli as the web requests are a little bit faster. I also discovered that some AMT versions have a defect on the TLS protocol so you definitely need to set the openssl settings or you would need to update AMT (I’m not sure how)
