Simplestreams mirror for custom images

Users
,
1

Hi,

I would like to setup a local custom images mirror for MaaS but Simplestreams repositories are … well … not that simple to setup.
I tried to find documentation describing the process to build such setup + tutorials on how to build custom images with the right format for SimpleStreams but there are very few things out there.
I already have some build scripts for building custom images based on debootstrap/chroot/ansible, I like it that way and can do about anything with this but how do I go from my custom image to a complete simplestreams repo ?

Any help gladly welcomed.

Regards.

2

Canonical provides LP:simplestreams to help create a stream but I agree it can be difficult. The MAAS team uses LP:maas-images to generate the public stream at images.maas.io. I integrated support for packer-maas when adding CentOS 8 to the stream. Below is what we use to generate the CentOS stream.

#!/bin/bash -ex

# Packer only recognizes upper case proxy environment variables - https://github.com/hashicorp/packer/issues/2275
export HTTP_PROXY="$http_proxy"
export HTTPS_PROXY="$https_proxy"

# meph2-import will use PACKER_PATH from the environment variable when defined, otherwise it searches $PATH
#export PACKER_PATH=$WORKSPACE_TMP/packer

# Cache ISOs into WORKSPACE_TMP so the cache survives multiple builds.
export PACKER_CACHE_DIR=$WORKSPACE_TMP

# Disable checking for a new Packer version, it's blocked anyway.
export CHECKPOINT_DISABLE=1

# Needed to enable serial output
export PACKER_LOG=1

sudo -En $WORKSPACE/maas-images/bin/meph2-import -u $([ "$FORCE" == "true" ] && echo "-f") packer-maas.yaml $WORKSPACE/centos-stream
sudo -En chown -R jenkins:jenkins $WORKSPACE/centos-stream
3

@Hybrid512, i feel your pain. as the tech author, i sometimes find i need to document other people’s stuff (like jq, for example) because the native product is more complex. to that end, can you give me an idea of the architecture you’re wanting to support with mirrors (i.e., size, machine count, number of subnets, etc.) ? it may help me produce a better MAAS-flavored simplestreams tutorial if i know something about the use cases. tia.

4

Hi @billwear, many thanks for your attention.

Well here is my use case scenario.

I intend to setup a CI/CD based on Gitlab/Gitlab-CI to build custom cloud images for our platform.
These images will be built with a custom script based on debootstrap/chroot/ansible mostly.
Once built, these images shall be pushed to a SimpleStreams repository that will be used for both MaaS (as a custom replacement repo instead of the maas.io repo) and for Openstack with the help of the Glance SimpleStreams sync Charm.
This repo will host ubuntu images but not only, it will also host VMWare ESX images and RHEL/Centos images.

So, my question is : how do you setup such repo ?
All the tutorials I found (and there aren’t much to say the least) talk about mirroring an existing repo, not building your own except one I found for LXD which more or less was doing the same thing that I want to achieve.

I see how the repo si organized and most things are quite easily “reverse engineerable” but there are things like GPG signatures and json metadata that are not that trivial.

Can you help me on that ? Could be the basis for a good tutorial on one of the Canonical blogs I guess.

Best regards.

5

@Hybrid512, clearly i dropped this ball. sorry. any progress, solutions? still in limbo?

6

Hi Bill, I see two issues with the existing blog entry at How to mirror images locally / https://maas.io/docs/how-to-mirror-images-locally:

  • a typo
MAAS will write the images to disk in the directory defined by the variable ‘IMAGE_DIR’ above, and the ‘location’ of the new boot source will be:

should be something like

sstream-mirror will write the images to disk in the directory defined by the variable ‘IMAGE_DIR’ above, and the ‘location’ of the new boot source will be:
  • it is missing the essential step: How to start the sstream service to serve those images? You mention URL=http://<myserver>/maas/images/ephemeral-v3/stable/ out of the blue, but what should happen with that URL? Use it as a custom images endpoint? But then again, how to start this http server?

Is there a chance that you could provide a working example how to operate simplestreams?

7

Linking @billwear explicitly, not sure if you get notified otherwise. Sorry for the noise.

8

Thanks for the feedback on the SimpleStreams mirroring documentation. I’ve logged an issue to review and improve it based on your comments. SimpleStreams isn’t really something we expect (or design for) users to be able to customise, so there’s that.

Maybe you can be more specific about “operate simplestreams”. Are you talking about setting up your own SimpleStreams server, or just can’t use the one we’ve got effectively?

9

The later, I just tried to run it as it was outlined in the documentation, and with the instructions given, I do not end up with a working system.

10

I am also very interested in this. I do not like uploading custom images to postgres.

11

images downloaded from simplestream go into postgres as well.
In MAAS 3.5 (will be released in few months) we moved the images from postgres to the disk

1 Like