Maas Enlistment fails on some of the Dell R650 nodes

[vallerul]

Hello,

I am seeing that some baremetal nodes, are failing during enlistment, after bringing up Enlistment image and during the run of commissioning scripts.

I see that the nodes turn off, after failing to run majority of the commissioning scripts.

Here is the errors that i see, in the logs:

2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: request to http://140.163.188.121:5240/MAAS/metadata/2012-03-01/ failed. sleeping 1
.: HTTP Error 401: Unauthorized
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: Traceback (most recent call last):
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1764, in
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: main()
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1745, in main
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: fail_count = run_scripts_from_metadata(
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1505, in run_sc
ripts_from_metadata
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: fail_count += run_scripts(
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1480, in run_sc
ripts
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: fail_count = run_serial_scripts(
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1270, in run_se
rial_scripts
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: send_unsent_results(scripts)
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: File “/tmp/user_data.sh.3xVPcy/bin/maas-run-remote-scripts”, line 1232, in send_u
nsent_results
2023-01-12T12:59:10-05:00 maas-enlisting-node cloud-init[2511]: args[“runtime”] = unsent_script[“runtime”]
2023-01-12T12:59:11-05:00 maas-enlisting-node cloud-init[2511]: KeyError: ‘runtime’
2023-01-12T12:59:11-05:00 maas-enlisting-node cloud-init[2511]: 2023-01-12 12:59:10,712 - cc_scripts_user.py[WARNING]: Failed to run module scripts
-user (scripts in /var/lib/cloud/instance/scripts)
2023-01-12T12:59:11-05:00 maas-enlisting-node cloud-init[2511]: 2023-01-12 12:59:10,713 - util.py[WARNING]: Running module scripts-user (<module ‘c
loudinit.config.cc_scripts_user’ from ‘/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py’>) failed
2023-01-12T12:59:11-05:00 maas-enlisting-node cloud-init[2511]: Cloud-init v. 22.4.2-0ubuntu0~20.04.2 finished at Thu, 12 Jan 2023 12:59:11 +0000.
Datasource DataSourceMAAS [http://140.163.188.121:5240/MAAS/metadata/]. Up 97.53 seconds
2023-01-12T12:59:11-05:00 maas-enlisting-node cloud-init: #############################################################
2023-01-12T12:59:11-05:00 maas-enlisting-node systemd[1]: cloud-final.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

May i know, what could be the cause for above?

Regards,
Lohit

[troyanov]

Hi @vallerul

Can you tell us a bit more about your setup and what MAAS version you are using?

[billwear]

@vallerul, can you give us more info about your setup and MAAS version? we really don’t have enough to go on, here.

[vallerul]

It turns out that the issue was related to Date/time on the servers. Correcting the Date/time on the BIOS of the servers, helped fixing the issue.