Hi,
Snap updated automatically maas from 3.3.4 to 3.3.5 on January 18th at 04:37 UTC. Since, I cannot access the MAAS dashboard and I’m also not able to access via the CLI:
maas02:~# maas login admin http://160.98.45.26:5240/MAAS
API key (leave empty for anonymous access):
The MAAS server gave an unexpected response: 502
maas02:~#
The OpenStack installed on the machines managed by MAAS work well and with a “maas status” on a terminal mentionned all services RUNNING except the proxy which was not started.
A reboot of the MAAS server was made yesterday and the “maas status” is eve worse than before, not only the proxy is stopped but also dhcpd, dhcpd6 and syslog:
maas02:~# maas status
bind9 RUNNING pid 1271, uptime 6:22:57
dhcpd STOPPED Not started
dhcpd6 STOPPED Not started
http RUNNING pid 1332, uptime 6:22:55
ntp RUNNING pid 1273, uptime 6:22:57
proxy STOPPED Not started
rackd RUNNING pid 1274, uptime 6:22:57
regiond RUNNING pid 1276, uptime 6:22:57
syslog STOPPED Not started
maas02:~#
I found errors in the logs at the time of the update but as a non specialist of MAAS, I do not know how to go further. Does already someone experience similar problem and if yes could it be corrected and how ?
In advance thanks for any help.
Here are hopefully useful information:
Current version of maas installed by snap: 3.3.5 stable
MAAS config:
maas02:~# maas config
Mode: region+rack
Settings:
maas_url=http://160.98.45.26:5240/MAAS
database_host=10.23.0.2
database_port=None
database_name=maasdb
database_user=maasuser
database_pass=(hidden)
maas02:~#
and here the content of log files at the time of the update
content of /var/snap/maas/common/log/maas.log
2024-01-18T04:20:24.506456+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.
2024-01-18T04:21:24.510681+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.]
2024-01-18T04:22:54.770904+00:00 maas02 maas.drivers.power.ipmi: [warn] using a non-secure cipher suite id
2024-01-18T04:22:55.117503+00:00 maas02 maas.drivers.power.ipmi: message repeated 7 times: [ [warn] using a non-secure cipher suite id]
2024-01-18T04:27:24.505983+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.
2024-01-18T04:27:54.500609+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.]
2024-01-18T04:28:09.665710+00:00 maas02 maas.drivers.power.ipmi: [warn] using a non-secure cipher suite id
2024-01-18T04:28:09.980391+00:00 maas02 maas.drivers.power.ipmi: message repeated 7 times: [ [warn] using a non-secure cipher suite id]
2024-01-18T04:29:24.488292+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.
2024-01-18T04:31:54.991122+00:00 maas02 maas.rpc.rackcontrollers: message repeated 5 times: [ [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.]
2024-01-18T04:33:24.387204+00:00 maas02 maas.import-images: [info] Starting rack boot image import
2024-01-18T04:33:24.391522+00:00 maas02 maas.import-images: [info] Downloading image descriptions from http://160.98.45.26:5240/MAAS/images-stream/streams/v1/index.json
2024-01-18T04:33:24.391597+00:00 maas02 maas.import-images: [info] Rack downloading image descriptions from 'http://160.98.45.26:5240/MAAS/images-stream/streams/v1/index.json'.
2024-01-18T04:33:24.699969+00:00 maas02 maas.import-images: [info] Finished importing boot images, the region does not have any new images.
2024-01-18T04:33:24.702480+00:00 maas02 maas.import-images: [info] Starting rack boot image import
2024-01-18T04:33:24.788587+00:00 maas02 maas.drivers.power.ipmi: [warn] using a non-secure cipher suite id
2024-01-18T04:33:25.162970+00:00 maas02 maas.drivers.power.ipmi: message repeated 7 times: [ [warn] using a non-secure cipher suite id]
2024-01-18T04:37:24.492181+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.
2024-01-18T04:37:54.768181+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.4-13189-g.f88272d1e has connected to region 'maas02'.]
2024-01-18T04:38:49.920193+00:00 maas02 maas.networks.monitor: [info] certificate-expiration-check: Process ID 273934 assumed responsibility.
2024-01-18T04:38:49.920202+00:00 maas02 maas.networks.monitor: [info] vault-secrets-cleanup: Process ID 273934 assumed responsibility.
2024-01-18T04:38:49.920206+00:00 maas02 maas.service_monitor: [info] Service 'maas-syslog' is not on, it will be started.
2024-01-18T04:38:49.920209+00:00 maas02 maas.dns: [error] Reloading BIND failed (is it running?): Command `rndc -c /var/snap/maas/32636/bind/rndc.conf.maas reload` returned non-zero exit status 1:#012rndc: connection to remote host closed.#012* This may indicate that the#012* remote server is using an older#012* version of the command protocol,#012* this host is not authorized to connect,#012* the clocks are not synchronized,#012* the key signing algorithm is incorrect#012* or the key is invalid.
2024-01-18T04:38:50.327175+00:00 maas02 maas.service_monitor: [info] Service 'bind9' is not on, it will be started.
2024-01-18T04:38:50.515238+00:00 maas02 maas.import-images: [info] Downloading image descriptions from http://images.maas.io/ephemeral-v3/stable/
2024-01-18T04:38:50.515526+00:00 maas02 maas.import-images: [info] Region downloading image descriptions from 'http://images.maas.io/ephemeral-v3/stable/'.
2024-01-18T04:38:50.642130+00:00 maas02 maas.import-images: [warn] Ignoring unsupported product com.ubuntu.maas.stable:1:pxelinux:pxe:amd64
2024-01-18T04:38:50.708082+00:00 maas02 maas.service_monitor: [error] Service 'bind9' failed to start. Its current state is 'dead' and 'Result: exit-code'.
2024-01-18T04:38:51.141232+00:00 maas02 maas.bootsources: [info] Updated boot sources cache.
2024-01-18T04:38:51.166648+00:00 maas02 maas.bootresources: [info] Started importing of boot images from 1 source(s).
2024-01-18T04:38:51.167415+00:00 maas02 maas.import-images: [info] Downloading image descriptions from http://images.maas.io/ephemeral-v3/stable/
2024-01-18T04:38:51.167463+00:00 maas02 maas.import-images: [info] Region downloading image descriptions from 'http://images.maas.io/ephemeral-v3/stable/'.
2024-01-18T04:38:51.270606+00:00 maas02 maas.import-images: [warn] Ignoring unsupported product com.ubuntu.maas.stable:1:pxelinux:pxe:amd64
2024-01-18T04:38:51.667876+00:00 maas02 maas.bootresources: [info] Importing images from source: http://images.maas.io/ephemeral-v3/stable/
2024-01-18T04:38:52.238217+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:ga-20.04 20240101
2024-01-18T04:38:52.294036+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:ga-20.04-lowlatency 20240101
2024-01-18T04:38:52.343882+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04 20240101
2024-01-18T04:38:52.391392+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-edge 20240101
2024-01-18T04:38:52.439846+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-lowlatency 20240101
2024-01-18T04:38:52.495203+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-lowlatency-edge 20240101
2024-01-18T04:38:52.545915+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:ga-22.04 20231222
2024-01-18T04:38:52.593132+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:ga-22.04-lowlatency 20231222
2024-01-18T04:38:52.645397+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:hwe-22.04 20231222
2024-01-18T04:38:52.690967+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:hwe-22.04-edge 20231222
2024-01-18T04:38:52.740910+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:hwe-22.04-lowlatency 20231222
2024-01-18T04:38:52.795043+00:00 maas02 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:22.04:amd64:hwe-22.04-lowlatency-edge 20231222
2024-01-18T04:38:58.756803+00:00 maas02 maas.service_monitor: [info] Service 'maas-syslog' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:38:58.758900+00:00 maas02 maas.service_monitor: [info] Service 'chrony' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:38:58.764513+00:00 maas02 maas.service_monitor: [info] Service 'maas-http' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:38:58.827345+00:00 maas02 maas.networks.monitor: [info] networks-monitoring: Process ID 273932 assumed responsibility.
2024-01-18T04:38:58.827359+00:00 maas02 maas.networks.monitor: [info] version-update-check: Process ID 273932 assumed responsibility.
2024-01-18T04:38:58.827362+00:00 maas02 maas.service_monitor: [info] Service 'maas-http' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:38:58.833194+00:00 maas02 maas.service_monitor: [info] Service 'maas-syslog' has been started and is 'running'.
2024-01-18T04:38:59.628488+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:00.101591+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.]
2024-01-18T04:39:00.160273+00:00 maas02 maas.refresh: [info] Refreshing rack controller hardware information.
2024-01-18T04:39:00.190468+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:00.354306+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:01.672998+00:00 maas02 maas.drivers.power.ipmi: [warn] using a non-secure cipher suite id
2024-01-18T04:39:02.222537+00:00 maas02 maas.drivers.power.ipmi: message repeated 7 times: [ [warn] using a non-secure cipher suite id]
2024-01-18T04:39:11.827525+00:00 maas02 maas.service_monitor: [info] Service 'maas-dhcpd' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:39:11.973474+00:00 maas02 maas.service_monitor: [info] Service 'maas-http' has been restarted. Its current state is 'on' and 'running'.
2024-01-18T04:39:16.778263+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:16.989779+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:22.905341+00:00 maas02 maas.refresh: [info] Refreshing rack controller hardware information.
2024-01-18T04:39:46.917424+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:39:47.176552+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.]
2024-01-18T04:40:16.166060+00:00 maas02 maas.bootresources: [info] Finished importing of boot images from 1 source(s).
2024-01-18T04:40:16.790158+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:40:18.067761+00:00 maas02 maas.rpc.rackcontrollers: message repeated 2 times: [ [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.]
2024-01-18T04:40:41.059428+00:00 maas02 maas.import-images: [info] Starting rack boot image import
2024-01-18T04:40:41.068577+00:00 maas02 maas.import-images: [info] Downloading image descriptions from http://160.98.45.26:5240/MAAS/images-stream/streams/v1/index.json
2024-01-18T04:40:41.068978+00:00 maas02 maas.import-images: [info] Rack downloading image descriptions from 'http://160.98.45.26:5240/MAAS/images-stream/streams/v1/index.json'.
2024-01-18T04:40:41.255688+00:00 maas02 maas.import-images: [info] Downloading boot resources from http://160.98.45.26:5240/MAAS/images-stream/streams/v1/index.json
2024-01-18T04:40:46.821369+00:00 maas02 maas.rpc.rackcontrollers: [info] Existing rack controller 'maas02' running version 3.3.5-13222-g.78dd996c0 has connected to region 'maas02'.
2024-01-18T04:41:03.481677+00:00 maas02 maas.import-images: [info] Writing boot image metadata.
2024-01-18T04:41:03.487534+00:00 maas02 maas.import-images: [info] Linking boot images snapshot /var/snap/maas/common/maas/boot-resources/snapshot-20240118-044041
2024-01-18T04:41:03.550474+00:00 maas02 maas.import-images: [info] Cleaning up old snapshots and cache.
2024-01-18T04:41:03.569963+00:00 maas02 maas.import-images: [info] Finished importing boot images.
2024-01-18T04:45:46.499599+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T04:50:16.498356+00:00 maas02 maas.power_monitor_service: message repeated 2 times: [ [error] Failed to query nodes' power status: ]
2024-01-18T04:50:56.570078+00:00 maas02 maas.dhcp.probe: [error] Unable to probe for DHCP servers:
2024-01-18T04:52:31.497768+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T04:54:46.499911+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T04:56:56.508054+00:00 maas02 maas.power_monitor_service: [error] Lost connection to region controller.
2024-01-18T04:59:01.496031+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:00:46.501537+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:00:56.557787+00:00 maas02 maas.dhcp.probe: [error] Unable to probe for DHCP servers:
2024-01-18T05:01:16.499023+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:05:46.498322+00:00 maas02 maas.power_monitor_service: message repeated 2 times: [ [error] Failed to query nodes' power status: ]
2024-01-18T05:05:46.519487+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:08:01.495901+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:10:16.497490+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:10:46.519494+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:10:56.579463+00:00 maas02 maas.dhcp.probe: [error] Unable to probe for DHCP servers:
2024-01-18T05:12:31.496551+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:14:46.498925+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:15:46.520933+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:17:01.496394+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:19:16.500332+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:20:46.521396+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:20:56.554643+00:00 maas02 maas.dhcp.probe: [error] Unable to probe for DHCP servers:
2024-01-18T05:21:31.497131+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:23:46.497641+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:25:46.502331+00:00 maas02 maas.boot_image_download_service: [error] Failed to download images:
2024-01-18T05:26:01.496437+00:00 maas02 maas.power_monitor_service: [error] Failed to query nodes' power status:
2024-01-18T05:30:31.496896+00:00 maas02 maas.power_monitor_service: message repeated 2 times: [ [error] Failed to query nodes' power status: ]
content of /var/snap/maas/common/log/rackd.log
2024-01-18 04:33:24 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency-edge:jammy: to_add=['20230328'] to_remove=[]
2024-01-18 04:33:24 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency:jammy: to_add=['20230328'] to_remove=[]
2024-01-18 04:33:24 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04:jammy: to_add=['20230328'] to_remove=[]
2024-01-18 04:33:54 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.
2024-01-18 04:37:04 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=43146, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:37:24 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37744, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:37:24 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=1431' authenticated.
2024-01-18 04:37:24 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=1431) with MAAS version 3.3.4-13189-g.f88272d1e.
2024-01-18 04:37:34 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=42512, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:37:54 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37724, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:37:54 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37726, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:37:54 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=1431' authenticated.
2024-01-18 04:37:54 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=1431' authenticated.
2024-01-18 04:37:54 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=1431) with MAAS version 3.3.4-13189-g.f88272d1e.
2024-01-18 04:37:55 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=1431) with MAAS version 3.3.4-13189-g.f88272d1e.
2024-01-18 04:38:04 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=58846, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 twisted.internet.unix: [info] (Port '/var/snap/maas/common/maas/dhcpd.sock' Closed)
2024-01-18 04:38:27 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:38:27 twisted.internet.defer: [critical]
Traceback (most recent call last):
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
result = current_context.run(
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/snap/maas/28521/lib/python3.10/site-packages/provisioningserver/rackdservices/lease_socket_service.py", line 121, in processNotification
yield pause(wait, self.clock)
builtins.AttributeError: 'LeaseSocketService' object has no attribute 'clock'
2024-01-18 04:38:27 provisioningserver.utils.services: [info] New interface monitoring state: {}
2024-01-18 04:38:27 provisioningserver.utils.services: [info] Stopping neighbour discovery for interfaces: {'ens224', 'ens192', 'ens160'}
2024-01-18 04:38:27 provisioningserver.utils.services: [info] Stopping beaconing for interfaces: {'ens224', 'ens192', 'ens160'}
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=53402, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=47288, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=38618, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=47304, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens192]: 0 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens192]: 215382 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens192]: 0 packets dropped by kernel
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens160]: 639068 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens160]: 854450 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens160]: 0 packets dropped by kernel
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=60292, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=38652, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=55294, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=57790, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens224]: 0 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens224]: 215382 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-beacons[ens224]: 0 packets dropped by kernel
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens160]: 1932145 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens160]: 1932145 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens160]: 0 packets dropped by kernel
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens192]: 2290553 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens192]: 2290553 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens192]: 0 packets dropped by kernel
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens224]: 32 packets captured
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens224]: 32 packets received by filter
2024-01-18 04:38:27 provisioningserver.utils.services: [info] observe-arp[ens224]: 0 packets dropped by kernel
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37726, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37744, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=46202, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37724, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=56368, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5251, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=35720, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5251, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=35730, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5251, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 provisioningserver.rpc.clusterservice: [info] Lost all connections to region controllers. Stopping service(s) dhcpd.
2024-01-18 04:38:27 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=44174, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5251, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:38:27 twisted.internet.defer: [critical]
Traceback (most recent call last):
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/base.py", line 492, in fireEvent
result = callable(*args, **kwargs)
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/application/service.py", line 284, in stopService
l.append(defer.maybeDeferred(service.stopService))
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 190, in maybeDeferred
result = f(*args, **kwargs)
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/application/service.py", line 284, in stopService
l.append(defer.maybeDeferred(service.stopService))
--- <exception caught here> ---
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 190, in maybeDeferred
result = f(*args, **kwargs)
File "/snap/maas/28521/lib/python3.10/site-packages/provisioningserver/rackdservices/lease_socket_service.py", line 68, in stopService
self.processor.stop()
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/task.py", line 213, in stop
assert self.running, "Tried to stop a LoopingCall that was " "not running."
builtins.AssertionError: Tried to stop a LoopingCall that was not running.
2024-01-18 04:38:27 twisted.scripts: [info] Server Shut Down.
2024-01-18 04:38:46 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:46 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:46 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:46 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:46 provisioningserver.utils.services: [info] Starting beaconing for interfaces: {'ens160', 'ens224', 'ens192'}
2024-01-18 04:38:46 provisioningserver.utils.services: [info] Beaconing process for ens160 started.
2024-01-18 04:38:46 provisioningserver.utils.services: [info] Beaconing process for ens224 started.
2024-01-18 04:38:46 provisioningserver.utils.services: [info] Beaconing process for ens192 started.
2024-01-18 04:38:47 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:47 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:47 provisioningserver.utils.services: [info] observe-beacons[ens160]: tcpdump: listening on ens160, link-type EN10MB (Ethernet), snapshot length 16384 bytes
2024-01-18 04:38:47 provisioningserver.utils.services: [info] observe-beacons[ens224]: tcpdump: listening on ens224, link-type EN10MB (Ethernet), snapshot length 16384 bytes
2024-01-18 04:38:48 provisioningserver.utils.services: [info] observe-beacons[ens192]: tcpdump: listening on ens192, link-type EN10MB (Ethernet), snapshot length 16384 bytes
2024-01-18 04:38:48 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:48 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:49 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:49 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:50 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:50 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:51 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:51 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:51 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:52 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:52 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:52 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:53 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:53 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:53 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:54 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:54 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:54 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:55 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:55 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:55 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:56 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:57 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:57 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:57 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:58 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:58 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:58 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://160.98.45.26:5240/MAAS)
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Making connections to event-loops: maas02:pid=274023, maas02:pid=274024, maas02:pid=274026, maas02:pid=274027
2024-01-18 04:38:59 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=34154, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5251, flowInfo=0, scopeID=0))
2024-01-18 04:38:59 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=45204, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:38:59 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=47740, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:38:59 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=55320, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274024' authenticated.
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274023' authenticated.
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Fully connected to all 4 event-loops on all 1 region controllers (maas02).
2024-01-18 04:38:59 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:00 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274024) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:00 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274027) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:00 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274023) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:01 provisioningserver.rackdservices.dhcp_probe_service: [info] Probe for external DHCP servers started on interfaces: ens160, ens192, ens224.
2024-01-18 04:39:01 provisioningserver.utils.services: [warn] Couldn't report test results: Unexpected error [Remote end closed connection without response]
2024-01-18 04:39:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=48146, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:39:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=48148, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:39:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:39:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:39:16 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:39:16 twisted.internet.defer: [critical]
Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\xa0b-\x1a', 98, 'Address already in use')
2024-01-18 04:39:16 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:39:16 twisted.internet.defer: [critical]
Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n\x17\x00\x02', 98, 'Address already in use')
2024-01-18 04:39:16 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:39:16 twisted.internet.defer: [critical]
Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n+\x00\x02', 98, 'Address already in use')
2024-01-18 04:39:16 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:17 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:28 provisioningserver.utils.services: [info] New interface monitoring state: {'ens160': {'neighbour': True, 'mdns': True}, 'ens192': {'neighbour': True, 'mdns': True}, 'ens224': {'neighbour': True, 'mdns': True}}
2024-01-18 04:39:28 provisioningserver.utils.services: [info] Starting neighbour discovery for interfaces: {'ens160', 'ens224', 'ens192'}
2024-01-18 04:39:28 provisioningserver.utils.services: [info] Neighbour observation process for ens160 started.
2024-01-18 04:39:28 provisioningserver.utils.services: [info] Neighbour observation process for ens224 started.
2024-01-18 04:39:28 provisioningserver.utils.services: [info] Neighbour observation process for ens192 started.
2024-01-18 04:39:28 provisioningserver.utils.services: [info] mDNS observation process started.
2024-01-18 04:39:29 provisioningserver.utils.services: [info] observe-arp[ens224]: tcpdump: listening on ens224, link-type EN10MB (Ethernet), snapshot length 64 bytes
2024-01-18 04:39:29 provisioningserver.utils.services: [info] observe-arp[ens160]: tcpdump: listening on ens160, link-type EN10MB (Ethernet), snapshot length 64 bytes
2024-01-18 04:39:29 provisioningserver.utils.services: [info] observe-arp[ens192]: tcpdump: listening on ens192, link-type EN10MB (Ethernet), snapshot length 64 bytes
2024-01-18 04:39:31 provisioningserver.rackdservices.dhcp_probe_service: [info] External DHCP probe complete.
2024-01-18 04:39:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=52984, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:39:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=53000, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:39:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=53002, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5250, flowInfo=0, scopeID=0))
2024-01-18 04:39:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:39:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:39:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274026' authenticated.
2024-01-18 04:39:47 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:47 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:39:47 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274026) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:40:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=42168, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:40:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=42172, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:40:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=42176, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='2001:620:40b:1045:250:56ff:fe92:b1c0', port=5252, flowInfo=0, scopeID=0))
2024-01-18 04:40:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274024' authenticated.
2024-01-18 04:40:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274024' authenticated.
2024-01-18 04:40:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274024' authenticated.
2024-01-18 04:40:17 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274024) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:40:18 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274024) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:40:19 provisioningserver.rpc.clusterservice: [info] Rack controller 'by6f8t' registered (via maas02:pid=274024) with MAAS version 3.3.5-13222-g.78dd996c0.
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-efi-signed:amd64:generic:uefi: to_add=['20230222.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-efi:arm64:generic:uefi: to_add=['20230222.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-ieee1275:ppc64el:generic:open-firmware: to_add=['20210509.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:pxelinux:i386:generic:pxe: to_add=['20210203.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-20.04-lowlatency:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-20.04:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-22.04-lowlatency:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-22.04:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-edge:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-lowlatency-edge:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-lowlatency:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-edge:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency-edge:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-efi-signed:amd64:generic:uefi: to_add=['20230222.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-efi:arm64:generic:uefi: to_add=['20230222.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:grub-ieee1275:ppc64el:generic:open-firmware: to_add=['20210509.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:pxelinux:i386:generic:pxe: to_add=['20210203.0'] to_remove=[]
2024-01-18 04:40:41 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-20.04-lowlatency:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33718, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:40:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33724, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:40:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33728, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:40:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:40:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:40:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:40:47 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-20.04:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:48 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-22.04-lowlatency:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:52 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-22.04:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:53 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-edge:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:55 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-lowlatency-edge:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:56 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04-lowlatency:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:57 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-20.04:focal: to_add=['20240101'] to_remove=[]
2024-01-18 04:40:58 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-edge:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:40:59 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency-edge:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:41:01 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04-lowlatency:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:41:02 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-22.04:jammy: to_add=['20231222'] to_remove=[]
2024-01-18 04:41:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=59364, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=59380, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=59382, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:41:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:41:16 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:41:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=55642, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=55648, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=55654, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:41:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:41:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:41:46 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas02:pid=274027' authenticated.
2024-01-18 04:42:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=39390, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=39402, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:16 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=39410, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:37 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:42:37 twisted.internet.defer: [critical]
Traceback (most recent call last):
--- <exception caught here> ---
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/lib/python3.10/site-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
failure.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
error.trap(RemoteAmpError)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
twisted.internet.defer.CancelledError:
2024-01-18 04:42:38 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:42:38 twisted.internet.defer: [critical]
Traceback (most recent call last):
--- <exception caught here> ---
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/lib/python3.10/site-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
failure.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
error.trap(RemoteAmpError)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
twisted.internet.defer.CancelledError:
2024-01-18 04:42:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33622, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33634, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:46 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=33646, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5253, flowInfo=0, scopeID=0))
2024-01-18 04:42:51 twisted.internet.defer: [critical] Unhandled error in Deferred:
2024-01-18 04:42:51 twisted.internet.defer: [critical]
Traceback (most recent call last):
--- <exception caught here> ---
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/lib/python3.10/site-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
failure.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
error.trap(RemoteAmpError)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
self.raiseException()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
raise self.value.with_traceback(self.tb)
twisted.internet.defer.CancelledError:
2024-01-18 04:42:52 twisted.internet.defer: [critical] Unhandled error in Deferred:
content of /var/snap/maas/common/log/regiond.log
2024-01-18 04:31:56 maasserver.dhcp: [info] Successfully configured DHCPv4 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:31:56 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:33:26 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas02' (by6f8t).
2024-01-18 04:33:26 maasserver.regiondservices.active_discovery: [info] Active network discovery: Unable to initiate network scanning on any rack controller. Verify that the rack controllers are started and have connected to the region.
2024-01-18 04:37:04 RegionServer,10914,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=43146, flowInfo=0, scopeID=0))
2024-01-18 04:37:04 maasserver.ipc: [info] Worker pid:1431 lost RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:37:24 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37744, flowInfo=0, scopeID=0))
2024-01-18 04:37:24 maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:10.43.0.2:37744'.
2024-01-18 04:37:25 maasserver.ipc: [info] Worker pid:1431 registered RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:37:26 maasserver.dhcp: [info] Successfully configured DHCPv4 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:37:26 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:37:34 RegionServer,10915,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=42512, flowInfo=0, scopeID=0))
2024-01-18 04:37:34 maasserver.ipc: [info] Worker pid:1431 lost RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:37:35 maasserver.dhcp: [info] Successfully configured DHCPv4 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:37:35 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:37:54 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37724, flowInfo=0, scopeID=0))
2024-01-18 04:37:54 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37726, flowInfo=0, scopeID=0))
2024-01-18 04:37:54 maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:10.43.0.2:37724'.
2024-01-18 04:37:54 maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:10.43.0.2:37726'.
2024-01-18 04:37:55 maasserver.ipc: [info] Worker pid:1431 registered RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:37:55 maasserver.ipc: [info] Worker pid:1431 registered RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:37:56 maasserver.dhcp: [info] Successfully configured DHCPv4 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:37:56 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:38:04 RegionServer,10916,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=58846, flowInfo=0, scopeID=0))
2024-01-18 04:38:04 maasserver.ipc: [info] Worker pid:1431 lost RPC connection to ('by6f8t', '10.43.0.2', 5252).
2024-01-18 04:38:05 maasserver.dhcp: [info] Successfully configured DHCPv4 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:38:05 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas02 (by6f8t)'.
2024-01-18 04:38:10 regiond: [info] 127.0.0.1 GET /MAAS/api/2.0/machines/?agent_name=c9a5d008-b9f8-4342-88e9-abc562edccaf&id=sexdbx&id=draqqd&id=684hws&id=atf7st&id=hbka8f&id=rwrawq&id=kekxp7&id=wqnsct HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2024-01-18 04:38:11 regiond: [info] 127.0.0.1 GET /MAAS/api/2.0/machines/?agent_name=c9a5d008-b9f8-4342-88e9-abc562edccaf&id=sexdbx&id=draqqd&id=684hws&id=atf7st&id=hbka8f&id=rwrawq&id=kekxp7&id=wqnsct HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2024-01-18 04:38:26 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas02' (by6f8t).
2024-01-18 04:38:26 maasserver.regiondservices.active_discovery: [info] Active network discovery: Unable to initiate network scanning on any rack controller. Verify that the rack controllers are started and have connected to the region.
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 IPCWorker,client: [info] IPCWorker connection lost (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/28521/maas-regiond.sock'))
2024-01-18 04:38:27 RegionServer,0,::ffff:160.98.45.26: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=53402, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,10842,::ffff:160.98.45.26: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=47288, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,10844,::ffff:160.98.45.26: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=38618, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 maasserver.rpc.regionservice: [info] Rack controller 'by6f8t' disconnected.
2024-01-18 04:38:27 RegionServer,10843,::ffff:160.98.45.26: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:160.98.45.26', port=47304, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 IPCWorker,client: [info] IPCWorker connection lost (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/28521/maas-regiond.sock'))
2024-01-18 04:38:27 RegionServer,10822,::ffff:10.23.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=60292, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,0,::ffff:10.23.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=38652, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,10821,::ffff:10.23.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=55294, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 maasserver.rpc.regionservice: [info] Rack controller 'by6f8t' disconnected.
2024-01-18 04:38:27 RegionServer,10823,::ffff:10.23.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=5253, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.23.0.2', port=57790, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 IPCWorker,client: [info] IPCWorker connection lost (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/28521/maas-regiond.sock'))
2024-01-18 04:38:27 RegionServer,10919,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37726, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,10917,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37744, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 RegionServer,0,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=46202, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 maasserver.rpc.regionservice: [info] Rack controller 'by6f8t' disconnected.
2024-01-18 04:38:27 RegionServer,10918,::ffff:10.43.0.2: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.43.0.2', port=37724, flowInfo=0, scopeID=0))
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 maasserver.workers: [info] Killing worker pid:1427.
2024-01-18 04:38:27 maasserver.workers: [info] Killing worker pid:1429.
2024-01-18 04:38:27 maasserver.workers: [info] Killing worker pid:1430.
2024-01-18 04:38:27 maasserver.workers: [info] Killing worker pid:1431.
2024-01-18 04:38:27 -: [info] Received SIGTERM, shutting down.
2024-01-18 04:38:27 -: [critical] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
self.runUntilCurrent()
--- <exception caught here> ---
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 260, in stop
super().stop()
File "/snap/maas/28521/usr/lib/python3/dist-packages/twisted/internet/base.py", line 708, in stop
raise error.ReactorNotRunning("Can't stop reactor that isn't running.")
twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.
2024-01-18 04:38:27 maasserver.ipc: [info] Worker pid:1430 IPC disconnected.
2024-01-18 04:38:28 maasserver.ipc: [info] Worker pid:1431 IPC disconnected.
2024-01-18 04:38:29 twisted.scripts: [info] Server Shut Down.
2024-01-18 04:38:46 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:46 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:47 maasserver.eventloop_273934.master: [info] Calling start_up to start region process
2024-01-18 04:38:47 maasserver.regiondservices.active_discovery: [info] Active network discovery: Discovery interval set to 10800 seconds.
2024-01-18 04:38:48 maasserver: [error] Error while calling ScanNetworks: Unable to get RPC connection for rack controller 'maas02' (by6f8t).
2024-01-18 04:38:48 maasserver.regiondservices.active_discovery: [info] Active network discovery: Unable to initiate network scanning on any rack controller. Verify that the rack controllers are started and have connected to the region.
2024-01-18 04:38:48 maasserver.listener: [info] Listening for database notifications.
2024-01-18 04:38:48 maasserver.region_controller: [info] Successfully configured proxy.
2024-01-18 04:38:49 maasserver.region_controller: [critical] Failed configuring DNS; killing and restarting
Traceback (most recent call last):
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- <exception caught here> ---
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/snap/maas/32636/lib/python3.10/site-packages/maasserver/region_controller.py", line 337, in _checkSerial
raise DNSReloadError(
maasserver.region_controller.DNSReloadError: Failed to reload DNS; timeout or rdnc command failed.
2024-01-18 04:38:49 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:49 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:49 maasserver.eventloop_274026.worker: [info] Calling start_up to start region process
2024-01-18 04:38:50 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:50 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:50 maasserver.eventloop_274023.worker: [info] Calling start_up to start region process
2024-01-18 04:38:50 -: [info] IPCWorker connection established (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock'))
2024-01-18 04:38:50 twisted.internet.protocol.Factory: [info] IPCMaster connection established (HOST:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock') PEER:UNIXAddress(None))
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274026 IPC connected.
2024-01-18 04:38:50 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:50 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:50 maasserver.eventloop_274024.worker: [info] Calling start_up to start region process
2024-01-18 04:38:50 twisted.scripts: [info] twistd 22.1.0 (/snap/maas/32636/bin/python3 3.10.12) starting up.
2024-01-18 04:38:50 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2024-01-18 04:38:50 maasserver.eventloop_274027.worker: [info] Calling start_up to start region process
2024-01-18 04:38:50 twisted.internet.protocol.Factory: [info] IPCMaster connection established (HOST:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock') PEER:UNIXAddress(None))
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274026 opened RPC listener on port:5250.
2024-01-18 04:38:50 -: [info] IPCWorker connection established (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock'))
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274023 IPC connected.
2024-01-18 04:38:50 twisted.internet.protocol.Factory: [info] IPCMaster connection established (HOST:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock') PEER:UNIXAddress(None))
2024-01-18 04:38:50 -: [info] IPCWorker connection established (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock'))
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274024 IPC connected.
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:1:bootloader-download/com.ubuntu.maas.stable:1:grub-efi-signed:uefi:amd64: to_add=['20231004.0'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:1:bootloader-download/com.ubuntu.maas.stable:1:grub-efi:uefi:arm64: to_add=['20231004.0'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:1:bootloader-download/com.ubuntu.maas.stable:1:grub-ieee1275:open-firmware:ppc64el: to_add=['20210509.0'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:1:bootloader-download/com.ubuntu.maas.stable:1:pxelinux:pxe:amd64: to_add=['20230615.0'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:1:bootloader-download/com.ubuntu.maas.stable:1:pxelinux:pxe:i386: to_add=['20210203.0'] to_remove=[]
2024-01-18 04:38:50 twisted.internet.protocol.Factory: [info] IPCMaster connection established (HOST:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock') PEER:UNIXAddress(None))
2024-01-18 04:38:50 -: [info] IPCWorker connection established (HOST:UNIXAddress(None) PEER:UNIXAddress('/var/snap/maas/32636/maas-regiond.sock'))
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:centos-bases-download/com.ubuntu.maas.stable:centos-bases:7.0:amd64: to_add=['20231216_01'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:centos-bases-download/com.ubuntu.maas.stable:centos-bases:8:amd64: to_add=['20211104_01'] to_remove=[]
2024-01-18 04:38:50 maasserver.region_controller: [critical] Failed to kill and restart DNS.
Traceback (most recent call last):
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
self._startRunCallbacks(fail)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
result = current_context.run(
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/snap/maas/32636/lib/python3.10/site-packages/provisioningserver/utils/service_monitor.py", line 432, in killService
state = yield self.ensureService(name)
File "/snap/maas/32636/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/snap/maas/32636/lib/python3.10/site-packages/provisioningserver/utils/service_monitor.py", line 741, in _ensureService
raise ServiceActionError(error_msg)
provisioningserver.utils.service_monitor.ServiceActionError: Service 'bind9' failed to start. Its current state is 'dead' and 'Result: exit-code'.
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274024 opened RPC listener on port:5252.
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274027 IPC connected.
2024-01-18 04:38:50 maasserver.ipc: [info] Worker pid:274027 opened RPC listener on port:5253.
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:12.04:amd64:hwe-p: to_add=['20170424.1'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:12.04:amd64:hwe-q: to_add=['20170424'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:12.04:amd64:hwe-r: to_add=['20170424'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:12.04:amd64:hwe-s: to_add=['20170424'] to_remove=[]
2024-01-18 04:38:50 sstreams: [info] com.ubuntu.maas:stable:v3:download/com.ubuntu.maas.stable:v3:boot:12.04:amd64:hwe-t: to_add=['20170424'] to_remove=[]