It started to work when I changed this in BMC:
Enable “RMCP+ Cipher Suite3 Configuration for each LAN channel” on the dedicated BMC interface
and
Setting Broad Compability on SSL Cipher.
The SSL certificate is still not valid (out of date) so MAAS seems to accept it as long as the other settings are changed.
Removing the Power MAC that the system sets may also be the solution. It is the MAC of the BMC interface MAAS activates not the dedicated interface that should be used.