Images are Always Queued for Download

contextualist · 19 July 2025 06:23

I have a fresh installation of MAAS from snap:

> snap list maas
Name  Version                  Rev    Tracking    Publisher   Notes
maas  3.5.7-16375-g.69525c9de  40008  3.5/stable  canonical✓  -

In addition, I am using postgresql 14 and I’m on Ubuntu 22.04.
I’m following the official set up guide. The only unusual setup I can think of is that we are using our own DHCP server, and I did disabled DHCP in MAAS.
I’m using the UI for setup and the images stuck at Queued for Download for hours.

I took a look at journalctl -u snap.maas.pebble.service --since "1 hour ago", and I thought the following might be noteworthy:

Jul 18 23:11:26 XXX maas-agent[3376836]: ERR Temporal worker pool failure error="Namespace default is not found."
Jul 18 23:11:27 XXX maas.pebble[3290660]: 2025-07-19T06:11:27.001Z [pebble] Service "agent" stopped unexpectedly with code 1
Jul 18 23:11:27 XXX maas.pebble[3290660]: 2025-07-19T06:11:27.001Z [pebble] Service "agent" on-failure action is "restart", waiting ~500ms before restart (backoff 1)
Jul 18 23:11:27 XXX maas.pebble[3290660]: 2025-07-19T06:11:27.536Z [pebble] Service "agent" starting: sh -c "exec systemd-cat -t maas-agent $SNAP/bin/run-maas-agent"

Attaching `maas status` for reference

> maas status
Service          Startup   Current   Since
agent            disabled  active    today at 06:21 UTC
apiserver        enabled   active    today at 01:50 UTC
bind9            disabled  active    today at 01:51 UTC
dhcpd            disabled  inactive  -
dhcpd6           disabled  inactive  -
http             disabled  active    today at 01:50 UTC
ntp              disabled  active    today at 01:50 UTC
proxy            disabled  inactive  today at 02:00 UTC
rackd            enabled   active    today at 01:50 UTC
regiond          enabled   active    today at 01:50 UTC
syslog           disabled  active    today at 01:50 UTC
temporal         disabled  active    today at 01:50 UTC
temporal-worker  disabled  active    today at 06:12 UTC

Please let me know if there are any places I should be checking. Thanks!

r00ta · 19 July 2025 08:22

Can you try to

sudo snap stop maas
sudo snap run --shell maas -c "maas-region dbupgrade"
sudo snap start maas
?

contextualist · 19 July 2025 17:02

Thank you @r00ta! I tried running the commands, and the issue persists.

> snap stop maas
2025-07-19T09:14:51-07:00 INFO Waiting for "snap.maas.pebble.service" to stop.
Stopped.
> snap run --shell maas -c "maas-region dbupgrade"
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  No migrations to apply.
Running Temporal migrations:
  Applied all migrations.
> snap start maas
Started.

When I mentioned clean installation earlier, I meant that I followed the instruction here and I removed the postgresql data directories.

I dug the log a bit more, and I noticed more information:

Here are the two errors that are maybe relevant before I performed the restart:

Jul 19 08:31:10 XXX maas-temporal-worker[3545640]: Task exception was never retrieved
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]: future: <Task finished name='Task-2' coro=<Worker.run() done, defined at /snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py:99> exception=RuntimeError('Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 110, kind: TimedOut, message: "Connection timed out" })))')>                                                                                                                                                                               
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]: Traceback (most recent call last):                                                                                                           
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 100, in run      
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     self._client = self._client or await get_client_async()                                                                
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 263, in wrapped
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     raise e
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 259, in wrapped                             
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     return await fn(*args, **kwargs)                                                                                                  
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 45, in get_client_async
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     return await Client.connect(                                                                                   
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/client.py", line 148, in connect               
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     await temporalio.service.ServiceClient.connect(connect_config),                                              
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 158, in connect
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     return await _BridgeServiceClient.connect(config)              
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 685, in connect
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     await client._connected_client()                 
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 698, in _connected_client
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     self._bridge_client = await temporalio.bridge.client.Client.connect(                                                                    
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:   File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/bridge/client.py", line 78, in connect
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]:     await temporalio.bridge.temporal_sdk_bridge.connect_client(                                                                          
Jul 19 08:31:10 XXX maas-temporal-worker[3545640]: RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 110, kind: TimedOut, message: "Connection timed out" })))                                                                                                                          
Jul 19 08:31:10 XXX maas.pebble[3290660]: 2025-07-19T15:31:10.676Z [pebble] Service "temporal-worker" stopped unexpectedly with code 0
Jul 19 08:31:10 XXX maas.pebble[3290660]: 2025-07-19T15:31:10.676Z [pebble] Service "temporal-worker" on-success action is "restart", waiting ~500ms before restart (backoff 1)                 
Jul 19 08:31:11 XXX maas.pebble[3290660]: 2025-07-19T15:31:11.191Z [pebble] Service "temporal-worker" starting: sh -c "exec systemd-cat -t maas-temporal-worker $SNAP/bin/run-temporal-worker"

Jul 19 09:01:53 XXX maas-regiond[3290834]: maasserver.bootresources: [critical] Importing boot resources failed.
Jul 19 09:01:53 XXX maas-regiond[3290834]:         Traceback (most recent call last):
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/usr/lib/python3.10/threading.py", line 953, in run
Jul 19 09:01:53 XXX maas-regiond[3290834]:             self._target(*self._args, **self._kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 821, in worker
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return target()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
Jul 19 09:01:53 XXX maas-regiond[3290834]:             task()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
Jul 19 09:01:53 XXX maas-regiond[3290834]:             task()
Jul 19 09:01:53 XXX maas-regiond[3290834]:         --- <exception caught here> ---
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             result = inContext.theWork()  # type: ignore[attr-defined]
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Jul 19 09:01:53 XXX maas-regiond[3290834]:             inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kw)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Jul 19 09:01:53 XXX maas-regiond[3290834]:             result = func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 137, in wrapper
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func_in_reactor(*args, **kwargs).wait(LONGTIME)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 198, in wait
Jul 19 09:01:53 XXX maas-regiond[3290834]:             result.raiseException()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Jul 19 09:01:53 XXX maas-regiond[3290834]:             raise self.value.with_traceback(self.tb)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Jul 19 09:01:53 XXX maas-regiond[3290834]:             current.result = callback(  # type: ignore[misc]
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/bootresources.py", line 938, in eb_import
Jul 19 09:01:53 XXX maas-regiond[3290834]:             failure.trap(DatabaseLockNotHeld)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Jul 19 09:01:53 XXX maas-regiond[3290834]:             self.raiseException()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Jul 19 09:01:53 XXX maas-regiond[3290834]:             raise self.value.with_traceback(self.tb)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             result = inContext.theWork()  # type: ignore[attr-defined]
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Jul 19 09:01:53 XXX maas-regiond[3290834]:             inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kw)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Jul 19 09:01:53 XXX maas-regiond[3290834]:             result = func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/orm.py", line 726, in call_with_connection
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/__init__.py", line 177, in call_with_lock
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return func(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/bootresources.py", line 1059, in _import_resources_with_lock
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return _import_resources_internal(notify)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/bootresources.py", line 1021, in _import_resources_internal
Jul 19 09:01:53 XXX maas-regiond[3290834]:             successful = download_all_boot_resources(
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/bootresources.py", line 885, in download_all_boot_resources
Jul 19 09:01:53 XXX maas-regiond[3290834]:             store.finalize(notify=notify)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/bootresources.py", line 671, in finalize
Jul 19 09:01:53 XXX maas-regiond[3290834]:             execute_workflow(
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/__init__.py", line 35, in wrapper
Jul 19 09:01:53 XXX maas-regiond[3290834]:             ret = asyncio.run(func(*args, **kwargs))
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return loop.run_until_complete(main)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return future.result()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/__init__.py", line 71, in execute_workflow
Jul 19 09:01:53 XXX maas-regiond[3290834]:             temporal_client = await get_client_async()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 263, in wrapped
Jul 19 09:01:53 XXX maas-regiond[3290834]:             raise e
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 259, in wrapped
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return await fn(*args, **kwargs)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 45, in get_client_async
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return await Client.connect(
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/client.py", line 148, in connect
Jul 19 09:01:53 XXX maas-regiond[3290834]:             await temporalio.service.ServiceClient.connect(connect_config),
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 158, in connect
Jul 19 09:01:53 XXX maas-regiond[3290834]:             return await _BridgeServiceClient.connect(config)
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 685, in connect
Jul 19 09:01:53 XXX maas-regiond[3290834]:             await client._connected_client()
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/service.py", line 698, in _connected_client
Jul 19 09:01:53 XXX maas-regiond[3290834]:             self._bridge_client = await temporalio.bridge.client.Client.connect(
Jul 19 09:01:53 XXX maas-regiond[3290834]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/temporalio/bridge/client.py", line 78, in connect
Jul 19 09:01:53 XXX maas-regiond[3290834]:             await temporalio.bridge.temporal_sdk_bridge.connect_client(
Jul 19 09:01:53 XXX maas-regiond[3290834]:         builtins.RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 110, kind: TimedOut, message: "Connection timed out" })))
Jul 19 09:01:53 XXX maas-regiond[3290834]:

After the restart, I also noticed some errors related to the DNS. We are running our own bind9 on the same machine. Not sure if that could cause troubles.

Jul 19 09:15:24 XXX maas-log[3562029]: maas.dns: [error] Freezing maas for update failed
Jul 19 09:15:24 XXX maas-log[3562029]: maas.dns: [error] Reloading BIND failed (is it running?): Command `rndc -c /var/snap/maas/40008/bind/rndc.conf.maas reload` returned non-zero exit status 1:#012rndc: connect failed: 127.0.0.1#954: connection refused
Jul 19 09:15:24 XXX maas-regiond[3561972]: maasserver.region_controller: [critical] Failed configuring DNS; killing and restarting
Jul 19 09:15:24 XXX maas-regiond[3561972]:         Traceback (most recent call last):
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
Jul 19 09:15:24 XXX maas-regiond[3561972]:             self._runCallbacks()
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Jul 19 09:15:24 XXX maas-regiond[3561972]:             current.result = callback(  # type: ignore[misc]
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
Jul 19 09:15:24 XXX maas-regiond[3561972]:             return _cancellableInlineCallbacks(gen)
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
Jul 19 09:15:24 XXX maas-regiond[3561972]:             _inlineCallbacks(None, gen, status)
Jul 19 09:15:24 XXX maas-regiond[3561972]:         --- <exception caught here> ---
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
Jul 19 09:15:24 XXX maas-regiond[3561972]:             result = current_context.run(gen.send, result)
Jul 19 09:15:24 XXX maas-regiond[3561972]:           File "/snap/maas/40008/lib/python3.10/site-packages/maasserver/region_controller.py", line 333, in _checkSerial
Jul 19 09:15:24 XXX maas-regiond[3561972]:             raise DNSReloadError(
Jul 19 09:15:24 XXX maas-regiond[3561972]:         maasserver.region_controller.DNSReloadError: Failed to reload DNS; timeout or rdnc command failed.
Jul 19 09:15:24 XXX maas-regiond[3561972]:
Jul 19 09:15:24 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-beacons[eno1]: tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 16384 bytes
Jul 19 09:15:24 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-beacons[ens2f0]: tcpdump: listening on ens2f0, link-type EN10MB (Ethernet), snapshot length 16384 bytes
Jul 19 09:15:24 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-beacons[ens2f1]: tcpdump: listening on ens2f1, link-type EN10MB (Ethernet), snapshot length 16384 bytes
Jul 19 09:15:24 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-beacons[eno2]: tcpdump: listening on eno2, link-type EN10MB (Ethernet), snapshot length 16384 bytes
Jul 19 09:15:24 XXX named[3562026]: 19-Jul-2025 09:15:24.727 starting BIND 9.18.30-0ubuntu0.22.04.2-Ubuntu (Extended Support Version) <id:>
Jul 19 09:15:24 XXX named[3562026]: 19-Jul-2025 09:15:24.727 running on Linux x86_64 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023

Jul 19 09:15:39 XXX maas-http[3562214]:  134.79.235.201 - - [19/Jul/2025:09:15:38 -0700] "POST /MAAS/metadata/2012-03-01/ HTTP/1.1" 200 2 "-" "Python-urllib/3.10"
Jul 19 09:15:39 XXX maas-http[3562214]:  134.79.235.201 - - [19/Jul/2025:09:15:39 -0700] "GET /MAAS/rpc/ HTTP/1.1" 200 432 "-" "provisioningserver.rpc.clusterservice.ClusterClientServi>
Jul 19 09:15:39 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-arp[ens2f0]: 4:15PM INF Service netmon started
Jul 19 09:15:39 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-arp[ens2f1]: 4:15PM INF Service netmon started
Jul 19 09:15:39 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-arp[eno2]: 4:15PM INF Service netmon started
Jul 19 09:15:39 XXX maas-regiond[3561972]: provisioningserver.utils.services: [info] observe-arp[eno1]: 4:15PM INF Service netmon started
Jul 19 09:15:39 XXX maas-regiond[3561972]: twisted.internet.defer: [critical] Unhandled error in Deferred:
Jul 19 09:15:39 XXX maas-regiond[3561972]: -: [info] b'/etc/resolv.conf' changed, reparsing
Jul 19 09:15:39 XXX maas-regiond[3561972]: -: [info] Resolver added ('127.0.0.53', 53) to server list
Jul 19 09:15:39 XXX maas-regiond[3561972]: twisted.internet.defer: [critical]
Jul 19 09:15:39 XXX maas-regiond[3561972]:         Traceback (most recent call last):
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
Jul 19 09:15:39 XXX maas-regiond[3561972]:             self.runUntilCurrent()
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
Jul 19 09:15:39 XXX maas-regiond[3561972]:             call.func(*call.args, **call.kw)
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 661, in callback
Jul 19 09:15:39 XXX maas-regiond[3561972]:             self._startRunCallbacks(result)
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
Jul 19 09:15:39 XXX maas-regiond[3561972]:             self._runCallbacks()
Jul 19 09:15:39 XXX maas-regiond[3561972]:         --- <exception caught here> ---
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Jul 19 09:15:39 XXX maas-regiond[3561972]:             current.result = callback(  # type: ignore[misc]
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
Jul 19 09:15:39 XXX maas-regiond[3561972]:             current_context.run(_inlineCallbacks, r, gen, status)
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1737, in _inlineCallbacks
Jul 19 09:15:39 XXX maas-regiond[3561972]:             status.deferred.callback(e.value)
Jul 19 09:15:39 XXX maas-regiond[3561972]:           File "/snap/maas/40008/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 660, in callback
Jul 19 09:15:39 XXX maas-regiond[3561972]:             assert not isinstance(result, Deferred)
Jul 19 09:15:39 XXX maas-regiond[3561972]:         builtins.AssertionError:
Jul 19 09:15:39 XXX maas-regiond[3561972]:

r00ta · 19 July 2025 17:56

Yes, bind9 on the same machine listening on port 53 is going to break things in different ways.

Logs of ‘maas-temporal’?

contextualist · 19 July 2025 18:18

I uploaded the output from journalctl -t maas-temporal --since "3 hour ago" to Ubuntu Pastebin

r00ta · 20 July 2025 08:37

Can you take a sos report and share it? https://maas.io/docs/how-to-report-and-review-bugs

contextualist · 20 July 2025 16:34

Here’s the sos report.