Controllers not displaying servers after update 3.2.1

I am running on ubuntu 20.04 and latest snap 3.2.1. We are running 2 rack and 2 region controllers. After doing snap refresh the controllers page doesnt get past loading. (screenshot) All the other pages work just fine. Machines, DNS, subnets images…

If I do a reboot of one of the rack controllers I do get an error telling me " One rack controller is not yet connected to the region. Visit the [rack controllers page] for more information."

but controllers page wont go past loading.

maas status shows all processes running
REGION
bind9 RUNNING pid 3018, uptime 2:19:55
http RUNNING pid 3247, uptime 2:19:51
ntp RUNNING pid 3286, uptime 2:19:51
proxy RUNNING pid 4333, uptime 2:19:41
regiond RUNNING pid 3021, uptime 2:19:55
syslog RUNNING pid 3251, uptime 2:19:51

RACK
bind9 RUNNING pid 2431, uptime 0:23:17
dhcpd RUNNING pid 2811, uptime 0:23:11
dhcpd6 STOPPED Not started
http RUNNING pid 14186, uptime 0:00:45
ntp RUNNING pid 2838, uptime 0:23:10
proxy RUNNING pid 14223, uptime 0:00:45
rackd RUNNING pid 2434, uptime 0:23:17
syslog RUNNING pid 14222, uptime 0:00:45

I did find this error in the log :/var/snap/maas/common/log/http/error.log

2022/08/04 05:16:22 [crit] 3025#3025: *13 connect() to unix:/var/snap/maas/23093/maas-regiond-webapp.sock failed (2: No such file or directory) while connecting to upstream, client: 172.30.0.34, server: , request: “GET /MAAS/rpc/ HTTP/1.1”, upstream: “http://unix:/var/snap/maas/23093/maas-regiond-webapp.sock:/MAAS/rpc/”, host: “172.30.0.30”
2022/08/04 05:16:22 [crit] 3025#3025: *13 connect() to unix:/var/snap/maas/23093/maas-regiond-webapp.sock failed (2: No such file or directory) while connecting to upstream, client: 172.30.0.34, server: , request: “GET /MAAS/rpc/ HTTP/1.1”, upstream: "http://unix:/var/snap/maas/23093/maas-regiond-webapp.sock

it looks like its missing folders?

Update: I think I have found the stack call error in :/var/snap/maas/common/log/regiond.log

when I refresh the webpage, this entry appears

2022-08-04 05:46:40 regiond: [info] 127.0.0.1 GET /MAAS/accounts/login/ HTTP/1.1 --> 200 OK (referrer: http://sv5-maas.avahc.com/MAAS/r/controllers; agent: Mozilla/5.0 (Windows NT
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36 Edg/103.0.1264.62)
2022-08-04 05:46:40 maasserver.websockets.protocol: [critical] Error on request (1) controller.list: None
        Traceback (most recent call last):
          File "/usr/lib/python3.8/threading.py", line 870, in run
            self._target(*self._args, **self._kwargs)
          File "/snap/maas/23093/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 821, in worker
            return target()
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
            task()
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
            task()
        --- <exception caught here> ---
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/23093/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/snap/maas/23093/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
            return func(*args, **kwargs)
          File "/snap/maas/23093/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 202, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/utils/orm.py", line 559, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.8/contextlib.py", line 75, in inner
            return func(*args, **kwds)
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/base.py", line 418, in prep_user_execute
            return self._call_method_track_queries(
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/base.py", line 442, in _call_method_track_queries
            result = method(params)
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/base.py", line 553, in list
            return [self.full_dehydrate(obj, for_list=True) for obj in objs]
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/base.py", line 553, in <listcomp>
            return [self.full_dehydrate(obj, for_list=True) for obj in objs]
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/base.py", line 233, in full_dehydrate
            return self.dehydrate(obj, data, for_list=for_list)
          File "/snap/maas/23093/lib/python3.8/site-packages/maasserver/websockets/handlers/controller.py", line 180, in dehydrate
            vlan_counts[self._vlans_ha[vlan_id]] += 1
        builtins.KeyError: None

I create a bug report for this

1 Like

Thanks, I was traveling out of office. Great to see quick response from team to correct as well.