Enlistment when using Custom repo's

Users
,
1

Hello.

We are currently using MAAS for deploying Ubuntu machines. We have run into a small problem that hopefully someone can assist with.

Currently we create our machines in MAAS with the API and then commission them. This has worked well, the machines commission and we can then deploy.

We need to move to a work flow where the machines are powered on and enlist themselves into MAAS, which we were hoping would be easy.

We can get the machines to boot and start the PXE image, but when enlisting and running the commissioning scripts, they are throwing the following errors, which breaks the enrolment process.

http://custom-repo/pulp/content/production/ubuntu jammy InRelease E: Repository has changed its ‘Origin’ value from Ubuntu to Pulp3

and

http://custom-repo/pulp/content/production/ubuntu jammy InRelease E: Repository has changed its ‘Label’ value from Ubuntu to ‘’

This is im sure caused by our use of Pulp for our custom repo’s.

this only happens during automatic enlistment, not during a commissioning.

Anyh thoughts?

2

Hi @srelf-eschercloudai ,

to confirm I understood correctly, you have configured MAAS to use custom repos in place of the official ubuntu ones (not just added them). Is that correct?

3

That is correct @ack

This is to allow us to pin to certain versions of known good packages, so we can control the upgrade cycle.

4

Can you please send the output of maas $profile package-repositories read and the full error output from the machine enlistment (sanitizing them as needed)?

5

Hi @ack

As requested

Exact error message, will have to be typed here, as the machine powers off and I have no way of getting to the error and nothing is logged into maas. but the gist is.

Failed to apply the settings specified in cloud-config

Intermixed with

cloud-init [pid] E: Repository http://my-repo-server.fqdn/pulp/content/production/ubuntu jammy InRelease Changed its Origin value from Ubuntu to Pulp 3

cloud-init [pid] E: Repository Repository http://my-repo-server.fqdn/pulp/content/production/ubuntu jammy InRelease Changed its label from Ubuntu to ""

This repeats for security-updates backports etc.

    {
        "name": "Ubuntu extra architectures",
        "url": "http://archive.ubuntu.com/ubuntu/",
        "distributions": [],
        "disabled_pockets": [],
        "disabled_components": [],
        "disable_sources": true,
        "components": [],
        "arches": [
            "armhf",
            "arm64",
            "ppc64el",
            "s390x"
        ],
        "key": "",
        "enabled": true,
        "id": 2,
        "resource_uri": "/MAAS/api/2.0/package-repositories/2/"
    },
    {
        "name": "Ubuntu archive",
        "url": "http://our-repo-server.fqdn/pulp/content/production/ubuntu",
        "distributions": [],
        "disabled_pockets": [],
        "disabled_components": [],
        "disable_sources": true,
        "components": [],
        "arches": [
            "amd64"
        ],
        "key": "-----BEGIN PGP PUBLIC KEY BLOCK----------END PGP PUBLIC KEY BLOCK-----",
        "enabled": true,
        "id": 1,
        "resource_uri": "/MAAS/api/2.0/package-repositories/1/"
    }
]

This is most likely an issue with PULP, as shown by https://github.com/microsoft/linux-package-repositories/issues/96 where M$ have had the same issue recently.

But what is odd is that commissioning works but enlistment doesn’t.

6

So something that @srelf-eschercloudai and I have picked up is that the cloud-init for generic and machine specific are very much deferent the way the sources are defined are not the same which seems to be why manual commissioning works while auto discovery commissioning fails.

failing example:
http://10...**:5248/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed

#cloud-config

apt:
  preserve_sources_list: false
  primary:
  - arches:
    - amd64
    - i386
    uri: http://repo_url/pulp/content/production/ubuntu
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu/
  proxy: http://10-**-**-0--23.maas-internal:8000/
  security:
  - arches:
    - amd64
    - i386
    uri: http://repo_url/pulp/content/production/ubuntu
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu/
  sources:
    default_key:
      key: '-----BEGIN PGP PUBLIC KEY BLOCK-----


        mQGNBGP8k6MBDADNnIGfvUK8FOn4Ob0+yFmDFG/wdZndkQDU2sPueB1qKvJLaTcD

        =Ozs1

        -----END PGP PUBLIC KEY BLOCK-----'
  sources_list: 'deb $PRIMARY $RELEASE main restricted multiverse universe

    # deb-src $PRIMARY $RELEASE main restricted multiverse universe

    deb $PRIMARY $RELEASE-updates main restricted multiverse universe

    # deb-src $PRIMARY $RELEASE-updates main restricted multiverse universe

    deb $PRIMARY $RELEASE-backports main restricted multiverse universe

    # deb-src $PRIMARY $RELEASE-backports main restricted multiverse universe

    deb $SECURITY $RELEASE-security main restricted multiverse universe

    # deb-src $SECURITY $RELEASE-security main restricted multiverse universe

    '

working example:
http://10...**:5248/MAAS/metadata/latest/by-id/br6cwh/?op=get_preseed

#cloud-config
apt:
  preserve_sources_list: false
  proxy: http://10.**.**.**:8000/
  sources:
    archive_key:
      key: '-----BEGIN PGP PUBLIC KEY BLOCK-----


        mQGNBGP8k6MBDADNnIGfvUK8FOn4Ob0+yFmDFG/wdZndkQDU2sPueB1qKvJLaTcD

        =Ozs1

        -----END PGP PUBLIC KEY BLOCK-----'
  sources_list: 'deb http://repo_url/pulp/content/production/ubuntu
    $RELEASE main restricted multiverse universe

    # deb-src http://repo_url/pulp/content/production/ubuntu
    $RELEASE main restricted multiverse universe

    deb http://repo_url/pulp/content/production/ubuntu $RELEASE-updates
    main restricted multiverse universe

    # deb-src http://repo_url/pulp/content/production/ubuntu
    $RELEASE-updates main restricted multiverse universe

    deb http://repo_url/pulp/content/production/ubuntu $RELEASE-security
    main restricted multiverse universe

    # deb-src http://repo_url/pulp/content/production/ubuntu
    $RELEASE-security main restricted multiverse universe

    deb http://repo_url/pulp/content/production/ubuntu $RELEASE-backports
    main restricted multiverse universe

    # deb-src http://repo_url/pulp/content/production/ubuntu
    $RELEASE-backports main restricted multiverse universe

    '

we also think that because the generic preseed has the default ubuntu source it’s confusing apt and that is why we get the origin error. should this default even be in the preseed if you have defined a custom source as the machine specific does not have this and works fine.

7

After a bit of digging through the deeper caverns of the internet and cobbling together information from a few different sources, i’ve come up with a workaround that works on Maas 3.5 at least.
Update the preseed templates on the region controllers to automatically allow the update when “apt-get update” is run, allowing the enlisting / commissioning to work.
(Not sure if the commissioning override is actually needed though.)

Place an “enlist” and a “commissioning” file in /var/snap/maas/current/preseeds with the following content

root@region:/# cat /var/snap/maas/current/preseeds/enlist
{{preseed_data}}
write_files:
  - path: /etc/apt/apt.conf.d/landscape-fix
    content: |
      Acquire::AllowReleaseInfoChange true;
root@region:/# cat /var/snap/maas/current/preseeds/commissioning
{{preseed_data}}
write_files:
  - path: /etc/apt/apt.conf.d/landscape-fix
    content: |
      Acquire::AllowReleaseInfoChange true;

References: