Image_cache_size documentation suggestion

Doc Requests
1

When looking for how to change the size of the image cache on rack controllers, I first found all of these documents which say that it is possible but don’t say how to do it:

MAAS | About images
MAAS | Reference: Release notes MAAS 3.5
Reference: Release notes MAAS 3.5 - Docs - Canonical MAAS | Discourse
Boot resources redesign - Docs - Canonical MAAS | Discourse
About images - Docs - Canonical MAAS | Discourse

I finally found it here:

How to use standard images (3.5-aware draft) - Docs - Canonical MAAS | Discourse

Searching for image_cache_size in the docs gives no results, suggesting that it’s not documented there at all:

MAAS Search “image_cache_size”

Would it be possible to add the explanation of how to change it somewhere in the docs, and maybe link to it from the “About images” and “Boot resource redesign” pages?

Thanks.

1 Like
2

It would also be good to have an explanation of how to do it with the snap install of MAAS.

3

Hi @andrew-boatrocker
This configuration parameter is indeed not very well documented, because we are not 100% sure if it will stay of will be removed.

In the future MAAS Agent will come with it’s own configuration file, but currently it is supervised by Rack Controller. Hence some of the configuration options that you can see here /var/snap/maas/current/agent.yaml can be modified through /var/snap/maas/current/rackd.conf

Maybe this code will help you a bit.

4

Should I be modifying /var/snap/maas/current/rackd.conf on the rack server or on the region server?

Will the service on the rack and/or region server need a restart after changing the file?

Thanks.

5

/var/snap/maas/current/rackd.conf on the Rack Controller server.
Yes, restart of rackd will be required.

6

Thanks. Currently /var/snap/maas/current/rackd.conf just has this in it:

maas_url: https://<dns name of our region server>:5443/MAAS

I assume I copy these bits over from /var/snap/maas/current/agent.yaml into rackd.conf and change them in order to override what’s in agent.yaml:

httpproxy:
  cache_dir: /var/snap/maas/common/maas/cache/httpproxy
  cache_size: 40426971955.2
7

@andrew-boatrocker sorry for a long response.

You should just add httpproxy_cache_size into /var/snap/maas/current/rackd.conf and restart MAAS. During restart a proper agent.yaml will be populated

2 Likes
8

Hi @troyanov Can i cache my custom image of size 8.3 Gb in rackd. If so please tell how to achieve it.

9

Hi @akashram611

You need to edit /var/snap/maas/current/rackd.conf (in case of a snap) and add a httpproxy_cache_size property there (value is in bytes).

Default cache size is set to 20GB, so it should cache your image just well. But keep in mind that thats an LRU cache, so your image can be evicted

10

In our case we have separate regiond and rackd. I also added this property httpproxy_cache_size: 20000000000 # 20GB in bytes
i can see the image is fetched from regiond on 2nd time deployment.
Regiond > Rackd (from iftop)
image

From Rackd to target machine
image

Please note that am using a custom image !

11

Hmm. Are you sure it downloads the full image? MAAS currently cannot cache the bootloader, so it there will be some traffic between machine and the regiond.

What is the size of a content located at:
/var/snap/maas/common/maas/cache/httpproxy/

What are the counters reported for httpproxy.* metrics under $rackd:5248/metrics/agent?

12

Hello,

this sounds like during migration from maas 3.4 to 3.5:

  • region will download images from database and save them as files on the region (exact location for deb format I was told it will be /var/lib/maas/image-storage on the maas region
  • rack will ‘sync’ with db which does not have images and delete them
  • rack will not re-download all images.

Can I pre-populate maas rack with images to avoid long synchronization times for large custom images ? (during upgrade from maas 3.4)

if the order of steps is incorrect, can someone from Canonical please provide exact order of the migration ?

For instance maas-rack could check the old maas rack cache for the file/checksum before attempting to download it from the region (and convert/move it to the new format/location).

Thank you!

13

Hello, Igor, long time no speak!

I think I understand your plight here, and we did come up with a workaround several years ago to pre-populate images on a remote rack controller before the rack itself was fully synchronized via the region controller’s point of view.

In this case however, in 3.5 and later, the images are no longer stored in the DB as they were previously, they’re stored on the filesystem itself.

For bootloader images, those are fetched on-demand from each machine via the region controller. They are not cached at all on the rack controller(s).

For non-bootloader images (eg: your custom images you would normally sync and upload to the region), those are cached on the rack controller in its own local cache. If the image is already present in the local rack controller cache, it’s served from there.

If it’s not already present when a machine asks for it, it’s fetched from the region controller, cached on the rack controller (in the rack’s filesystem, not the DB), and served from there. Any machine after the first ‘cold’ fetch of that image, will then pick up the copy that is now local to the rack in its cache.

What I think you’re asking for sounds the same as the request from back in 2021, where you want to ‘seed’ the rack controller with all of the images it might need, before that rack is fully joined to the region, and before the very first machine on that rack requests an image.

Is that a correct assumption?

You could, if you wanted to discuss specifics, file a case with the Support team, and we can engage more fully “internally”. We’re limited as to what we can share about your environment and its needs out here in the public Discourse forum.

Hope that helps!

1 Like