DHCP server will dead if network flapping

Hi guys,

I running maas 3.5 I notice the dhcp server will dead if network flapping in few second.
Here is the rack log with debug options.

Nov 20 08:26:39 rack-server-2 systemd-networkd[152]: maas-conn: Link DOWN
Nov 20 08:26:39 rack-server-2 systemd-networkd[152]: maas-conn: Lost carrier
Nov 20 08:26:39 rack-server-2 networkctl[3036560]: Interface "maas-conn" not found.
Nov 20 08:26:39 rack-server-2 systemd[1]: networkd-dispatcher.service: Got notification message from PID 3036560, but reception only permitted for main PID 162
Nov 20 08:26:39 rack-server-2 networkd-dispatcher[162]: ERROR:Failed to get interface "maas-conn" status: Command '['/usr/bin/networkctl', 'status', '--no-pager', '--no-legend', '--', 'maas-conn']' returned non-zero exit status 1.
Nov 20 08:26:40 rack-server-2 networkd-dispatcher[162]: WARNING:Unknown index 30 seen, reloading interface list
Nov 20 08:26:40 rack-server-2 systemd-networkd[152]: maas-conn: Link UP
Nov 20 08:26:40 rack-server-2 systemd-networkd[152]: maas-conn: Gained carrier
Nov 20 08:26:40 rack-server-2 systemd-networkd[152]: maas-conn: Gained IPv6LL
Nov 20 08:26:46 rack-server-2 squid[332]: 2024/11/20 08:26:46 kid1| Logfile: opening log stdio:/var/spool/squid/netdb.state
Nov 20 08:26:46 rack-server-2 squid[332]: 2024/11/20 08:26:46 kid1| Logfile: closing log stdio:/var/spool/squid/netdb.state
Nov 20 08:26:46 rack-server-2 squid[332]: 2024/11/20 08:26:46 kid1| NETDB state saved; 1 entries, 0 msec
Nov 20 08:26:49 rack-server-2 (squid-1): 1732091209.246      0 79.110.49.33 NONE_NONE/000 0 - error:transaction-end-before-headers - HIER_NONE/- -
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-dhcpd' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-dhcpd6' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'chrony' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-http' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'bind9' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-proxy' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-syslog' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:26:51 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-agent' is SERVICE_STATE.ON and 'running'.
Nov 20 08:26:51 rack-server-2 rackd[2985756]: tftp.protocol: [debug] TFTP Listener started at 10.96.253.4:69
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271027
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271027
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53252, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53266, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271027
Nov 20 08:27:01 rack-server-2 rackd[2985756]: message repeated 5 times: [ provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271027]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271028
Nov 20 08:27:01 rack-server-2 rackd[2985756]: message repeated 2 times: [ provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271028]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53280, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53286, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53296, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53304, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=53314, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=44140, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33556, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rackdservices.external: [critical] Failed to get external services configurations.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011Traceback (most recent call last):
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011--- <exception caught here> ---
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/external.py", line 517, in _tryUpdate
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    config = yield self._getConfiguration()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/external.py", line 489, in _getConfiguration
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    controller_type = yield client(
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    failure.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    error.trap(RemoteAmpError)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    self.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011twisted.internet.error.ConnectionDone: Connection was closed cleanly.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: Connection was closed cleanly.; interfaces: {'cloudagent': {'type': 'bridge', 'mac_address': '6a:b1:a9:18:35:79', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'cni-podman0': {'type': 'bridge', 'mac_address': '2a:06:94:85:34:4b', 'links': [{'mode': 'static', 'address': '10.88.0.1/16'}], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'cni0': {'type': 'bridge', 'mac_address': '0a:f2:c7:bc:d3:d3', 'links': [{'mode': 'static', 'address': '10.128.130.1/24'}, {'mode': 'static', 'address': 'fdff:ffff:ffff:ffff::1/112'}], 'enabled': True, 'parents': ['veth1dde6471', 'veth3adf1cba', 'veth835391b8', 'vethc761b244'], 'source': 'machine-resources', 'monitored': False}, 'dnsmasq': {'type': 'physical', 'mac_address': 'ea:76:ee:60:e9:5f', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'enp61s0f0': {'type': 'physical', 'mac_address': 'b0:44:14:e0:9c:7c', 'links': [{'mode': 'static', 'address': '<hide>', 'gateway': '<hide>'}, {'mode': 'static', 'address': '10.97.5.2/24'}, {'mode': 'static', 'address': '<hide>'}], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'enp61s0f1': {'type': 'physical', 'mac_address': 'b0:44:14:e0:9c:7d', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'flannel.1': {'type': 'physical', 'mac_address': '7e:90:98:5b:ea:e4', 'links': [{'mode': 'static', 'address': '10.128.130.0/32'}], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'gollum': {'type': 'physical', 'mac_address': '0e:1b:2e:4d:fd:a1', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'netdata': {'type': 'physical', 'mac_address': '56:f0:f2:f0:ad:ff', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'nodelocaldns': {'type': 'physical', 'mac_address': '2e:9e:3f:61:5d:6f', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'nodelocaldns-s': {'type': 'physical', 'mac_address': '36:22:97:c8:c7:9b', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}, 'veth1dde6471': {'type': 'physical', 'mac_address': '36:2f:a7:1b:f3:5e', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'veth3adf1cba': {'type': 'physical', 'mac_address': '82:b4:23:15:c7:86', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'veth835391b8': {'type': 'physical', 'mac_address': 'b6:dd:4b:01:f9:87', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'vethc761b244': {'type': 'physical', 'mac_address': '06:33:25:03:a2:95', 'links': [], 'enabled': True, 'parents': [], 'source': 'machine-resources', 'monitored': True}, 'yurthub-dummy0': {'type': 'physical', 'mac_address': '0e:19:e7:c2:f0:3a', 'links': [], 'enabled': False, 'parents': [], 'source': 'machine-resources', 'monitored': False}}
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011Traceback (most recent call last):
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011--- <exception caught here> ---
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1074, in do_action
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    yield self._updateInterfaces(interfaces)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1197, in _updateInterfaces
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    yield maybeDeferred(self._configureNetworkDiscovery, interfaces)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1394, in _configureNetworkDiscovery
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    monitoring_state = yield maybeDeferred(self.getDiscoveryState)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    failure.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    error.trap(RemoteAmpError)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    self.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011twisted.internet.error.ConnectionDone: Connection was closed cleanly.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33572, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=46998, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271028
Nov 20 08:27:01 rack-server-2 rackd[2985756]: message repeated 4 times: [ provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271028]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271029
Nov 20 08:27:01 rack-server-2 rackd[2985756]: message repeated 8 times: [ provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271029]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271030
Nov 20 08:27:01 rack-server-2 rackd[2985756]: message repeated 7 times: [ provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: region-server:pid=3271030]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33584, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=47008, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=47018, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=47038, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=47032, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33110, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33118, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33122, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33134, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rackdservices.service_monitor_service: [critical] Failed to monitor services and update region.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011Traceback (most recent call last):
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011--- <exception caught here> ---
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/service_monitor_service.py", line 92, in _updateRegion
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    yield client(
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/provisioningserver/rpc/common.py", line 149, in _global_intercept_errback
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    failure.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    current.result = callback(  # type: ignore[misc]
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    error.trap(RemoteAmpError)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    self.raiseException()
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011    raise self.value.with_traceback(self.tb)
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011twisted.internet.error.ConnectionDone: Connection was closed cleanly.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: #011
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33144, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=33160, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=50412, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=50428, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=50444, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=52102, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=52118, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=52134, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=52142, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=39336, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=39358, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=39342, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Lost all connections to region controllers. Stopping service(s) dhcpd,dhcpd6.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:27:01 rack-server-2 rackd[2985756]: ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.5', port=39372, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl stop maas-dhcpd
Nov 20 08:27:01 rack-server-2 maas-log[2986557]: maas.drivers.power.ipmi: message repeated 25 times: [ [warn] using a non-secure cipher suite id]
Nov 20 08:27:01 rack-server-2 maas-log[2986557]: maas.service_monitor: [info] Service 'maas-dhcpd' is not off, it will be stopped.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:01 rack-server-2 maas-log[2986557]: maas.service_monitor: [info] Service 'maas-dhcpd6' is not off, it will be stopped.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl stop maas-dhcpd6
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-http' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'chrony' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'bind9' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-proxy' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-syslog' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 systemd[1]: Stopping MAAS instance of ISC DHCP server for IPv4...
Nov 20 08:27:01 rack-server-2 systemd[1]: maas-dhcpd.service: Deactivated successfully.
Nov 20 08:27:01 rack-server-2 systemd[1]: Stopped MAAS instance of ISC DHCP server for IPv4.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-agent' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl stop maas-dhcpd
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:01 rack-server-2 systemd[1]: Stopping MAAS instance of ISC DHCP server for IPv6...
Nov 20 08:27:01 rack-server-2 systemd[1]: maas-dhcpd6.service: Deactivated successfully.
Nov 20 08:27:01 rack-server-2 systemd[1]: Stopped MAAS instance of ISC DHCP server for IPv6.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl stop maas-dhcpd6
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '3' from cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:01 rack-server-2 maas-log[2986557]: maas.service_monitor: [info] Service 'maas-dhcpd' has been stopped and is 'dead'.
Nov 20 08:27:01 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '3' from cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:01 rack-server-2 maas-log[2986557]: maas.service_monitor: [info] Service 'maas-dhcpd6' has been stopped and is 'dead'.
Nov 20 08:27:02 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=ActivityWorker
Nov 20 08:27:02 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Making connections to event-loops: region-server:pid=3271027, region-server:pid=3271028, region-server:pid=3271029, region-server:pid=3271030
Nov 20 08:27:02 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47958, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:02 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=55488, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:02 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:03 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:03 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=7bmcyr@agent:power WorkerID=7bmcyr@agent:2986558 WorkerType=ActivityWorker
Nov 20 08:27:03 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=7bmcyr@agent:main WorkerID=7bmcyr@agent:2986558 WorkerType=WorkflowWorker
Nov 20 08:27:03 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:03 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:03 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=55502, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:03 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=WorkflowWorker
Nov 20 08:27:03 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47974, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:03 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=55516, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:03 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47990, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:03 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=55522, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:03 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:03 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=ActivityWorker
Nov 20 08:27:03 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=ActivityWorker
Nov 20 08:27:03 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:04 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=7bmcyr@agent:main WorkerID=7bmcyr@agent:2986558 WorkerType=ActivityWorker
Nov 20 08:27:04 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=48000, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:04 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:04 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=WorkflowWorker
Nov 20 08:27:04 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=55532, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:04 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:04 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=agent:power@vlan-16 WorkerID=7bmcyr@agent:2986558 WorkerType=WorkflowWorker
Nov 20 08:27:04 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:05 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:05 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=41308, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:05 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=48780, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:05 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:05 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:05 rack-server-2 maas-agent[2986558]: WRN Failed to poll for task. Error="context deadline exceeded" Namespace=default TaskQueue=7bmcyr@agent:power WorkerID=7bmcyr@agent:2986558 WorkerType=WorkflowWorker
Nov 20 08:27:05 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=41312, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:05 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=48784, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:05 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=41316, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:06 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=48786, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5251, flowInfo=0, scopeID=0))
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:06 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:07 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=41324, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5250, flowInfo=0, scopeID=0))
Nov 20 08:27:07 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271028' authenticated.
Nov 20 08:27:07 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271027' authenticated.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271028) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:08 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271027) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:21 rack-server-2 maas-log[2986557]: maas.drivers.power.ipmi: [warn] using a non-secure cipher suite id
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor executing cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '3' from cmd: sudo --non-interactive systemctl status maas-dhcpd
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-dhcpd' is SERVICE_STATE.OFF and 'dead'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '3' from cmd: sudo --non-interactive systemctl status maas-dhcpd6
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-dhcpd6' is SERVICE_STATE.OFF and 'dead'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status chrony
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'chrony' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-http
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-http' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status bind9
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'bind9' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-proxy
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-proxy' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-syslog
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-syslog' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] [try:1] Service monitor got exit code '0' from cmd: sudo --non-interactive systemctl status maas-agent
Nov 20 08:27:21 rack-server-2 rackd[2985756]: provisioningserver.utils.service_monitor: [debug] Service 'maas-agent' is SERVICE_STATE.ON and 'running'.
Nov 20 08:27:26 rack-server-2 kernel: [3127845.977338] enp61s0f1 speed is unknown, defaulting to 1000
Nov 20 08:27:32 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop region-server:pid=3271029 (::ffff:172.28.0.95:5252): User timeout caused connection failure.
Nov 20 08:27:32 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop region-server:pid=3271030 (::ffff:172.28.0.95:5253): User timeout caused connection failure.
Nov 20 08:27:32 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47320, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:32 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47294, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:32 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:32 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:33 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:33 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:33 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47310, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:33 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47322, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:33 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47316, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:33 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47324, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:33 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47318, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:33 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:33 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:34 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:34 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47326, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:34 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:34 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=47320, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:34 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:35 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:35 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=57738, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:35 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=46418, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:35 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:35 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:35 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=57740, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:35 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=46430, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:35 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=57754, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:36 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=46432, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5253, flowInfo=0, scopeID=0))
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:36 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:37 rack-server-2 rackd[2985756]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.96.253.4', port=57770, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.97.0.1', port=5252, flowInfo=0, scopeID=0))
Nov 20 08:27:37 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271030' authenticated.
Nov 20 08:27:37 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Event-loop 'region-server:pid=3271029' authenticated.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271030) with MAAS version 3.5.1-16317-g.409891638.
Nov 20 08:27:38 rack-server-2 rackd[2985756]: provisioningserver.rpc.clusterservice: [info] Rack controller '7bmcyr' registered (via region-server:pid=3271029) with MAAS version 3.5.1-16317-g.409891638.

It seem that after rack reconnect to the region, the region will ask the rack to shutdown the dhcp server…

I think it maybe some problem about the ipc manager ?

Hi there, so it is normal for dhcpd to be restarted upon rackd reconnecting to the region-controller, as it will fetch new configuration and need to reload. What isn’t normal is those RPC connections dropping. Can you confirm if there is/was a network partition when this was observed? Are the region controller and rack controller on separate hosts or the same host?

I am not sure what you mean by “network partition.” We just restarted the switch, and the interface went down and came back up.

The rack and region controllers are not on the same machine.

I have conducted some tests here:
a. If I stop the rack during a network flap and wait for the region controller to consider the rack as dead, then restart the rack, the DHCP works normally.
b. The DHCP on the rack will never recover after a network flap unless I restart the region controller.
c. I have modified the IPCMaster code. I added a step where the region controller unregisters the rack RPC before re-registering it. Although this results in a lot of UPDATE and DELETE operations in the database, the rack can still run DHCP even when the interface goes down and up.
d. I noticed that the status of a rack remains in the region for a long time (4–5 minutes) after the rack has been disconnected.
e. Even if I only restart the rack without any network flapping, sometimes the DHCP service fails after the rack starts up and never runs again until the region controller is restarted.

Also there is a dhcp config generate bug: dhcp v6 dns need ipv6 also, If my region only have v4 and rack have v6, the dhcp v4 can not start, because the DNS options in v6 config will include a v4 ip and this is not allow for the dhcpd.

Hey @boxjan

Thank you so much for the investigations! Could you open bugs according to this doc https://maas.io/docs/how-to-report-and-review-bugs ? Please open one standalone bug for each of the issues and provide all the steps to reproduce them in details

How can I open a PR, for the dhcp v6 , I have a fix.

See https://documentation.ubuntu.com/launchpad/en/latest/how-to/contributing-changes/ and https://code.launchpad.net/maas .

You have to first sign the CLA https://ubuntu.com/legal/contributors .

Also, please open a bug as well for the dhcp issue so that the history has more context about what you are fixing

CLA need Canonical Project Manager or contact, I don’t known that.

You can put me jacopo dot rota at canonical dot com

bugs is here

for the problem 2089224, I have create a merge is here Merge into master : master : lp:~cjl-cn/maas : Git : Code : MAAS