Clear cache? (/var/snap/maas/common/maas/boot-resources/cache)

I’ve got an issue quite similar to this one from a few years ago -

Essentially, my cache is now about 20GB and bumping into my file system size. It looks like that’s causing the rack sync to hang.

I suspect that it’s not deleting/removing obsolete images from the cache, even if I bin them from the webui.

Is there a way to clean the cache?

Kind regards,
Jeff Berry, MRC Cognition and Brain Sciences Unit

Hi Jeff,

3.5 has a new mechanism to handle the images between the regions and racks. It will solve your issue by design, so I’d suggest you to upgrade to 3.5

Excellent. I’m on 3.3.9, so I’ll give that try. Thanks …

Jeff

I upgraded to 3.5.2 and my images (both larger than 2GB) were hanging at Queued for Download. I rebuilt /var to give the snap more room to work, but they’re still hanging.

The webui is up and running, and the snap connections look OK, I think.

Interface              Plug                        Slot                    Notes
avahi-observe          maas:avahi-observe          :avahi-observe          -
content                -                           maas:maas-logs          -
content                maas:test-db-socket         -                       -
hardware-observe       maas:hardware-observe       :hardware-observe       -
home                   maas:home                   :home                   -
kernel-module-observe  maas:kernel-module-observe  :kernel-module-observe  -
mount-observe          maas:mount-observe          :mount-observe          -
network                maas:network                :network                -
network-bind           maas:network-bind           :network-bind           -
network-control        maas:network-control        :network-control        -
network-observe        maas:network-observe        :network-observe        -
snap-refresh-control   maas:snap-refresh-control   :snap-refresh-control   -
system-observe         maas:system-observe         :system-observe         -
time-control           maas:time-control           :time-control           -

But when I try to load an image, it times out with [Errno 111] Connection refused.
There’s something else funny going on, I think. Since I rebuilt and moved /var, the logs in /var/snap/maas/common/log aren’t being written - they may be being written to the regular syslog.

The snap was throwing some apparmor errors, but setting it to complain didn’t do the trick. I’m not terribly familiar with snap - did moving /var break the snap somehow, or is there something else going on?

wdym?

Could you df -h on your region controller? Also, if you could extract the regiond logs (see here) we might understand if there is anything wrong

Surely -

Filesystem             Size  Used Avail Use% Mounted on
tmpfs                   26G  3.0M   26G   1% /run
efivarfs               304K  193K  107K  65% /sys/firmware/efi/efivars
/dev/mapper/vg0-lv--0  373G   42G  312G  12% /
tmpfs                  126G  1.1M  126G   1% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
tmpfs                  126G     0  126G   0% /run/qemu
/dev/mapper/vg0-lv--1   20G  183M   19G   1% /boot
/dev/mapper/vg0-lv--4  4.9G  735M  3.9G  16% /var/log
/dev/sda1              1.1G  6.2M  1.1G   1% /boot/efi
/dev/mapper/vg0-lv--2   15G  5.3G  8.7G  38% /home
/dev/mapper/vg0-lv--5  2.0G   48K  1.8G   1% /var/tmp
/dev/mapper/vg0-lv--3   20G   11G  7.7G  59% /var.old
/dev/sda2              1.5T   86G  1.4T   6% /install
tmpfs                   26G   16K   26G   1% /run/user/1000

So /var is now just part of / with oodles of room.

The regiond logs look like they’re proxying through pebble since the upgrade - is that the default for 3.5, btw? Here’s a pull from syslog . I can pull more if needed. There’s a ‘lost burst connection’ error showing up.

Nov 27 13:02:04 lsr-cluster-01 maas-regiond[4504]: RegionServer,9,::ffff:172.31.120.3: [info] RegionServer connec
tion lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:IPv6Ad
dress(type='TCP', host='::ffff:172.31.120.3', port=42500, flowInfo=0, scopeID=0))
Nov 27 13:02:04 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 lost burst connection t
o ('172.31.120.3', 5252).
Nov 27 13:02:04 lsr-cluster-01 maas-regiond[4504]: RegionServer,10,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=42506, flowInfo=0, scopeID=0))
Nov 27 13:02:04 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 lost burst connection t
o ('172.31.120.3', 5252).
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=52780, flowInfo=0, scopeID=0))
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=52782, flowInfo=0, scopeID=0))
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:52780'.
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Connection 39a32cb5-9abe-
44c0-a278-59d708534db7 is trusted and ready to respond/serve commands.
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:52782'.
Nov 27 13:02:24 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Connection f588ceba-d23b-
4be9-9929-c744ced8a754 is trusted and ready to respond/serve commands.
Nov 27 13:02:25 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 registered RPC connecti
on to ('d6dgg8', '172.31.120.3', 5252).
Nov 27 13:02:26 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 registered RPC connecti
on to ('d6dgg8', '172.31.120.3', 5252).
Nov 27 13:02:34 lsr-cluster-01 maas-regiond[4504]: RegionServer,11,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=42924, flowInfo=0, scopeID=0))
Nov 27 13:02:34 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 lost burst connection t
o ('172.31.120.3', 5252).
Nov 27 13:02:34 lsr-cluster-01 maas-regiond[4504]: RegionServer,12,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=42934, flowInfo=0, scopeID=0))
Nov 27 13:02:34 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 lost burst connection t
o ('172.31.120.3', 5252).
Nov 27 13:02:35 lsr-cluster-01 maas-regiond[4500]: regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (
referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=55176, flowInfo=0, scopeID=0))
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5252, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=55182, flowInfo=0, scopeID=0))
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:55176'.
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Connection 7aed6894-9d2d-
446e-a971-6be1972fdfa9 is trusted and ready to respond/serve commands.
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:55182'.
Nov 27 13:02:54 lsr-cluster-01 maas-regiond[4504]: maasserver.rpc.regionservice: [info] Connection e5aaae5d-8cb7-
4a3d-a0fe-222d27c8e45b is trusted and ready to respond/serve commands.
Nov 27 13:02:56 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4504 registered RPC connecti
on to ('d6dgg8', '172.31.120.3', 5252).
Nov 27 13:03:34 lsr-cluster-01 maas-regiond[4506]: RegionServer,10,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=54106, flowInfo=0, scopeID=0))
Nov 27 13:03:34 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 lost burst connection t
o ('172.31.120.3', 5253).
Nov 27 13:03:34 lsr-cluster-01 maas-regiond[4506]: RegionServer,11,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=54114, flowInfo=0, scopeID=0))
Nov 27 13:03:34 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 lost burst connection t
o ('172.31.120.3', 5253).
Nov 27 13:03:54 lsr-cluster-01 maas-regiond[4506]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=58066, flowInfo=0, scopeID=0))
Nov 27 13:03:54 lsr-cluster-01 maas-regiond[4506]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=58074, flowInfo=0, scopeID=0))
Nov 27 13:03:55 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:58074'.
Nov 27 13:03:55 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Connection 75e59673-b975-
48a1-a30e-8942889ac771 is trusted and ready to respond/serve commands.
Nov 27 13:03:55 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 registered RPC connecti
on to ('d6dgg8', '172.31.120.3', 5253).
Nov 27 13:03:56 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:58066'.
Nov 27 13:03:56 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Connection 89ab068e-16e8-
4492-9467-ade990ff8f46 is trusted and ready to respond/serve commands.
Nov 27 13:03:56 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 registered RPC connecti
on to ('d6dgg8', '172.31.120.3', 5253).
Nov 27 13:04:04 lsr-cluster-01 maas-regiond[4506]: RegionServer,12,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=57598, flowInfo=0, scopeID=0))
Nov 27 13:04:04 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 lost burst connection t
o ('172.31.120.3', 5253).
Nov 27 13:04:04 lsr-cluster-01 maas-regiond[4506]: RegionServer,13,::ffff:172.31.120.3: [info] RegionServer conne
ction lost (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:IPv6A
ddress(type='TCP', host='::ffff:172.31.120.3', port=57600, flowInfo=0, scopeID=0))
Nov 27 13:04:04 lsr-cluster-01 maas-regiond[4467]: maasserver.ipc: [info] Worker pid:4506 lost burst connection t
o ('172.31.120.3', 5253).
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=33934, flowInfo=0, scopeID=0))
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: twisted.internet.protocol.Factory: [info] RegionServer connect
ion established (HOST:IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=5253, flowInfo=0, scopeID=0) PEER:
IPv6Address(type='TCP', host='::ffff:172.31.120.3', port=33948, flowInfo=0, scopeID=0))
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:33934'.
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Connection 9c79fbe1-4dba-
42c0-89b8-6b9bb6304e70 is trusted and ready to respond/serve commands.
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Rack controller authentic
ated from '::ffff:172.31.120.3:33948'.
Nov 27 13:04:24 lsr-cluster-01 maas-regiond[4506]: maasserver.rpc.regionservice: [info] Connection 91f0e45c-8cf9-
4ea0-a2f9-2561a8512700 is trusted and ready to respond/serve commands.
'''

(And thank you!)

These are looking good. How many regions and racks do you have in your environment? What images are stuck in the Downloading state? Have you tried to stop/start the download process in the UI?

Just one controller -region+rack. Two images stuck downloading, both custom .tgz. One 2.3 GB, one 2.9GB. I was wondering about the size, because a couple of <1GB images did sync.

If I delete the big one via the webUI, it disappears. Then if I kick it off from the CLI with ‘maas admin boot-resources,’ it shows up again as ‘Queued for download’ and then … it worked this time.

Bloody hell. I wonder if the apparmor was causing the issue? When I tried earlier, it failed with the err 111 timeout. But maybe I didn’t try again after disabling apparmor on the snap.

Slightly different problem - releasing machines is now failing, sort of. They are being released, but marked as release failed. If I mark them broken, and then as fixed, I can deploy to them.

 Wed, 27 Nov. 2024 14:57:32	Node status event - 'cloudinit' running modules for final
 Wed, 27 Nov. 2024 14:57:32	Marking node failed - Failed to release machine.
 Wed, 27 Nov. 2024 14:57:32	Node changed status - From 'Releasing' to 'Releasing failed'
 Wed, 27 Nov. 2024 14:57:32	Node status event - 'cloudinit' running config-scripts_user with frequency once-per-instance
 Wed, 27 Nov. 2024 14:57:30	Script result - wipe-disks changed status from 'Installing dependencies' to 'Running'
 Wed, 27 Nov. 2024 14:57:30	Script result - wipe-disks changed status from 'Running' to 'Passed'
 Wed, 27 Nov. 2024 14:57:30	User releasing node
 Wed, 27 Nov. 2024 14:57:30	Releasing

This fails after a redeploy and rerelease. There’s a workaround, as I say, but it’s odd.

I’d say to look at the regiond logs as there might be additional info about the failure

Looks like it might be the fact that we’ve got a self-signed certificate. At least, there’s a ‘SSL: CERTIFICATE_VERIFY_FAILED’ error. Releasing was working in 3.3.9 with this certificate, and the UI is still working with https. Deploys are working, and there’s a workaround for release.

I suppose I’d better check comissioning.

FOLLOWUP - Commissioning seems to work.