Managing releases

[billwear]

About release management

Release management involves everything required to make a release available to external users. This work includes a mix of automation, manual activites, verification, and a good bit of waiting. Also, RC releases and final releases each have some specific, unique complications, which are covered toward the end this document.

And, on top of that, there are lots of little “gotchas” that will make things work weird, until you know to do them. We’ll try to catch as much of that as we can.

In the maas-release-tools repo, under the bin directory, you will find a script named release-status. This is where you know where you are in the release process and what to do next.

Start with checking out the commit of the MAAS repo that you want to release. From that working tree, run release-status $version_to_release. You’ll now see a bunch of checks, most of them being red. Start with the top most error and resolve that. Then run it again and resolve the top most error again.

The goal is that release-status should tell you exactly what to do. It doesn’t yet, but feel free to improve it as you go along. Especially one thing it doesn’t tell you are the stages that the release goes through. Read more about it below.

In most cases, this script will lead you through the steps you need to execute to build a release; however, you may need to get through the first few steps in this process before release-status becomes useful.

For tracking purposes, add a card to the team sprint board with the release version (e.g. 3.4.1). Our stakeholders, e.g. SEG/Support who may be waiting for a specific bugfix release and need to keep customers informed, will watch this card to know if the release moves to the consecutive sprint.

MAAS release stages

When releasing MAAS you go through a series of stages. You start by building the release binaries and then you push them through to different PPAs and snap channels, with some level of validation done between stages.

It’s important that you only build the binaries to be released once. You should never rebuild the release in the middle, you should always copy around the binaries only.

Release preparation

PPA	Snap
`you/maas-$major_version-next`	`$major_version/beta/release-prep`

This is where you build the initial binaries. For the snap, the revisions should already be built and published, most likely in the edge channel.

You’re working in your own PPA and in a snap branch since this is a very early stage of the release process. Here you do, at least, some basic smoke tests to ensure that the binaries you built can be installed and is working after installation.

After a smoke test, you can move on to the next stage.

Quality assurance

PPA	Snap
`maas/$major_version-next`	`$major_version/beta`

After you are confident that the release is good, it’s time to move the binaries to a more public place. At this stage, only basic QA has been done, so users that want to try out the release should be aware that it might be broken.

At this stage you should tell the Solutions QA team to test the release. Send an e-mail to their list, and CC the MAAS mailing list, so that the whole team is aware of the progress.

After Solutions QA have given their blessing, you can make the release official and move on to the final stage.

Public release

PPA (Beta)	Snap (Beta)
`maas/$major_version-next`	`$major_version/beta`

PPA (RC)	Snap (RC)
`maas/$major_version-next`	`$major_version/candidate`

PPA (Stable)	Snap (Stable)
`maas/$major_version`	`$major_version/stable`

The final step is to copy the binaries to the release PPA and snap channel. Note that the PPA and snap channel depend on whether it’s a beta, RC or stable release.

Now you tag the commit you release with the version number and send out the release announcement.

How to create a MAAS release

To produce a MAAS release, you’ll need to either execute the steps in this section or get someone to do them for you. Generally, you only need help if you don’t have the required access (true for very few of these steps), so you’d need to get someone on the MAAS core team to do them for you.

This section will tell you:

• How to create repositories.
• How to check system tests.
• How to version a release.
• How to make a build container.
• How to create a development environment.
• How to build the release.
• How to upload to the test PPA
• How to upload the PPA.
• How to confirm the snap build.
• How to create a milestone for the release.
• How to mark closed bugs as released.
• How to post a release.

This section gives a generic procedure for releasing MAAS. Note that there are unique complications for both RC releases and final releases, so be sure to consult the reference sections at the end of this document.

How to create repositories

When publishing a major release for the first time, create the appropriate repositories:

• Create a PPA on the MAAS launchpad page:
  – If you’re publishing Beta releases, you’ll want to use a PPA named “x.y-next” (e.g., “3.1-next”), with a descriptive name of “MAAS x.y test builds” (e.g., “MAAS 3.1 test builds”).
  – If you’re publishing a final release, you’ll want to use a PPA named “x.y” (e.g., “3.1”), with a descriptive name of “MAAS 3.0.”

• Create a test PPA on your personal launchpad page; this will be used to test your builds before pushing them up to a public PPA.

• Copy additional dependencies from the PPA of the previous major release to this new PPAs:
  – Go to the latest deps page.
  – Select “View package details” in the upper right to get here.
  – Towards the mid-upper right, select “Copy packages”.
  – Under “Packages”, in the right middle of the screen, select the correct OS series and click “Filter”.
  – On the screen that comes up, select the checkboxes next to all of the available packages.
  – Just below this list, select the Destination PPA to be each of those you have created above, in turn.
  – Below this, you can select either “Build the copied sources” or “Copy existing binaries”. In most cases, “copy existing binaries” should work fine.
  – Click “Copy packages” and make sure that the results confirm the copy; if not, fix the issue and try again.

• Request a new track on the snap store for the maas snap:
  – Go to the snapstore discourse forum.
  – If you’re not signed up, you’ll need to do that, first (FYI, there’s a confirmation step associated with sign up).
  – Request the new track with a message similar to this one, minus the middle paragraph about requesting 2.10 last week. Be sure to include the two bullet points about the snap IDs, and set the tag to store-requests.
  – If you’re a new snapcraft forum user, you’ll need to turn the links in the two ID bullet points into literals by surrounding them with backticks, as you will have limited ability to post links.
  – Pay attention to the forum for a response. It may take some time, in some cases.
  – Make sure that the maas-test-db snap uses (1) the same base as the maas snap, and (2) the latest Postgres version for that base.

[note]
Getting a new track on the snap store is a long-lead-time item; be sure to request it well in advance of your planned first release date.

How to check system tests

Both of these tests are scanning master, so if you’re starting from there, you really want both of these to be green at this moment:

– Check http://maas-integration-ci.internal:8080/job/maas-system-tests/ (for deb)
– Check http://maas-integration-ci.internal:8080/job/maas-system-tests-snap/ (for snap)

If you’re releasing a Beta, it’s worthwhile to find out why the tests are failing, e.g., “Is there some retooling going on which is causing the tests to fail, even though the code is fine?” You may be able to move forward anyway.

Before going further, you must make sure that the system tests pass for the revision you’re planning on releasing, for both the normal and -ea variant.

How to version a release

Run the maas-tag-release jenkins job to update the MAAS version.

1. Login to Jenkins at that URL
2. Click Build with parameters
3. Enter the details, paying particular attention to the version

This step will leave you a new commit in the MAAS master branch, versioned with the new release identifier.

Some important notes:

• The version should use the format ..(a|b|rc). For example, 3.1.0-Beta 2 would be labeled as “3.1.0b2” when tagging the release.

• The DEBFULLNAME should be e.g. Joe Bloggs, and DEBEMAIL e.g. joe.bloggs@canonical.com.

• There must be a GPG key registered in Launchpad that’s associated with the DEBFULLNAME and DEBEMAIL that you enter.

• Note that you cannot have a comment associated with your GPG key, as this will sometimes cause the signing tool to fail when building the release later.

How to make a build container

Create the build container

Run the following script – from the directory containing your MAAS git repo (but not the repo itself) – to automate creation of a build container:

#!/bin/bash
set -e

SRC=$(readlink -f "${1:-$PWD}")

if [ ! -d "$SRC" ]; then {
    echo "Expected '$1' to be a directory" >&2
    exit 1
}
fi

PROJECT_NAME=$(basename "$SRC")
CONTAINER_NAME=${CONTAINER:-$PROJECT_NAME}
SSH_ME=${SSH_ME:-$(bzr lp-login)}

SERIES=${SERIES:-focal}
# TODO: Conditionalise launching and setup based on existing or not
# lxc list --fast --format=json "$PROJECT_NAME" | grep -q "$PROJECT_NAME" 
lxc launch --verbose --debug ubuntu:$SERIES "$CONTAINER_NAME" -c raw.idmap="both $UID 1000" -c user.user-data="\
#cloud-config
{'ssh_import_id': '$SSH_ME', 'apt_pipelining': 'os', 'package_update': true,
 'packages': ['language-pack-en'], 'write-files': {'content': 'export DEBEMAIL=\"Adam Collard <adam.collard@canonical.com>\"', 'path': '/home/ubuntu/.bashrc', 'append': 'true'}
}
"

lxc exec "$CONTAINER_NAME" -- cloud-init status --wait || echo 'Old cloud-init, snoozing instead'; sleep 5
[ -f ~/.gitconfig ] && lxc file push ~/.gitconfig "$CONTAINER_NAME"/home/ubuntu/.gitconfig
if host "$CONTAINER_NAME".lxd >/dev/null; then
    DST="$CONTAINER_NAME".lxd
else
    DST=$(lxc list "$CONTAINER_NAME" -c4 --format=csv | cut -d' ' -f1)
fi
ssh-keyscan -H $DST >> ~/.ssh/known_hosts 2>/dev/null
lxc config device add "$CONTAINER_NAME" "${PROJECT_NAME}-src" disk source="$SRC" path=/home/ubuntu/"$PROJECT_NAME"
echo "$CONTAINER_NAME ready to hack on; try 
ssh ubuntu@$DST"

To run this script successfully, you’ll need to use an invocation something like this:

SERIES=<os-series-name> CONTAINER=<memorable-name> SSH_ME=<your-launchpad-ssh-id> hack-on

As an example only:

SERIES=focal CONTAINER=maas SSH_ME=billwear hack-on

Mount your gnupg directory

When the container is ready, you’ll also want to bind mount your ~/.gnupg directory, containing the key set you want to use to sign the release, to the container, like this:

lxc config device add maas gnupg-home disk source=/home/$USERNAME/.gnupg/ path=/home/ubuntu/.gnupg

Load your ssh directory

You might also want to copy the contents of your SSH keys to the container – excluding the authorized keys file – so that you can access git.launchpad.net easily.

Checkout your versioned commit

Note that your MAAS repo will be mounted automatically into the container as part of the hack-on process. You want to update that repo to the version you just committed with the Jenkins script.

How to create a development environment

Depending on whether you’ve built MAAS before, you may or may not have all the development tools you’ll need.

How to download the build tools

You’ll may want to run the command:

apt install build-essential

before you attempt to go any further; otherwise, you won’t have all the tools you need to build.

How to download the dependencies you’ll need

As with most build processes, you may need to install the source dependencies with a command like this:

make install-dependencies

Note that there’s a hiccup point during the installation of snapd, so you may hang for some time on this line:

if [ -x /usr/bin/snap ]; then cat required-packages/snaps | xargs -L1 sudo snap install; fi

It may even fail once or twice, depending on your internet connection, the phases of the moon, or what you had for dinner last night. Just try again.

Dealing with other missing packages

You may have to install one or more other packages as you go. Pay close attention to any errors: When the build fails, try entering the missing command to find out what package you need sudo apt install... to get going again. This may happen three or four times. This is normal, but pay attention.

How to build the release

Building the release is fairly straightforward:

    utilities/release-build $ubuntu_release

where $ubuntu_release is the series you’re building, e.g., focal.

Here are some important hints for the build:

• Make sure that DEBFULLNAME and DEBEMAIL variables are set to the right values for the debian package, e.g., the name and e-mail that matches the GPG key of the person who is building the release. If you think you may be in and out of your build container (likely), it’s probably best to set these two variables in your .bashrc file for the ubuntu user:

export DEBFULLNAME="your gpg full name"
export DEBEMAIL="your gpg email"

Make sure that you export them.

• Note that you can’t run the build in an su’d session: GPG has a bug that will cause the signing part to fail at the end; e.g., you can’t log on as root and then su - to your account – you have to sign on directly as the account that owns the correct GPG key, or the ubuntu account, if you’re using the hack-on script above.

• It is instinctive to set the $ubuntu_release to the MAAS release if you’re in a hurry. If the utilities/release-build $ubuntu_release returns a cryptic error like this one:

Unsupported distro: 2.8.7

then you should go back and set $ubuntu_release to the actual OS series, such as “bionic” or “focal.”

• There are two or three places where you may bomb out if your internet connection hiccups at the wrong time. These generally involve URLs. If this happens, just try it again; eventually, you will get all the way through the build.

How to upload to the test PPA

Upload the packages built in the previous step to the PPA, using the following script:

    utilities/release-upload <PPA> <changes file>

Set the as follows:

• When uploading a stable release (whether a final or a point release), a stable ppa:maas/M.m PPA should be used:

    utilities/release-upload ppa:maas/2.X build_pkg/maas_2.X.*.changes

The stable PPA can also be used for preleases of a M.m.0 release, since there’s no stable release for that major at this point

• When uploading a pre-release for a point release (e.g. M.m.1-rc1), a testing ppa:maas/M.m-next PPA should be used:

utilities/release-upload ppa:maas/2.X-next build_pkg/maas_2.X.*.changes

2. When Launchpad has finished the build, release it to the PPA.

How??

How to confirm the snap build

Confirm the snap builds, by following this procedure:

How??

n. When the snaps are built, release them to the right risk channel.

How to create a new milestone

Create a new, active milestone for the next release (after this one), to target bugs that weren’t completed for this release:

How to create a new milestone

We manually tag a release in git once the release has been built and published. To do so run the following in the maas repo:

git checkout $RELEASED_COMMIT
git tag $RELEASE_VERSION
git push --tags 

How??

How to mark closed bugs as released

Use bin/release-manage in maas-release-tools to mark closed bugs as released:

    bin/release-manage maas move-done-bugs next $milestone
    bin/release-manage maas release-milestone $milestone

Also, create a release for the milestone on LP, with the following procedure:

How??

How to post a final release

If this is a final release, you should also complete the following steps:

1. Post the release notes to discourse (or see that they are posted, if this is the normal responsibility of another teammate).

2. Post an announcment in discourse, or see that one is posted.

3. Switch the default snap track to the new release by:

How??

There are other complications for a final release, as shown below.

How to push the MAAS test DB to the new snap track

When starting a new major release, you should update or push the maas-test-db snap to the new track, with the following procedure:

How??

RC release reference

When an RC has been released, a new branch should be created, and build scripts should be updated to use the new branch. The following procedures are necessary:

1. Use the update-release-branch job to cut the new branch, following the <Major version>.<Minor version> naming scheme

2. Ask the design team to branch maas-ui

3. Ask @billwear to branch lp:maas-offline-docs

4. Point the maas-ui and lp:maas-offline-docs git submodules in lp:maas to the M.m branch:

cd maas/src/maas-offline-docs/src
git pull
git checkout M.m
maas/src/maasui/src
git pull
git checkout M.m

6. Commit the changes:

git commit src/maas-offline-docs/src src/maasui/src -m "Switch docs and UI to M.m branch"

7. change the MAAS_PPA value in Makefile to the release PPA (e.g. ppa:maas/M.m)

8. Set the Source: for the Snap builds to the branch you just created when building the Snap with Launchpad.

9. Release snap builds to candidate channel (and close beta).

Final release reference

On final releases of a new major, some extra steps are required:

1. Release snap builds to the stable channel (and close candidate).
2. Release maas-test-db to the stable channel (and close candidate).
3. Switch default tracks for both maas and maas-test-db to the new version.

Troubleshooting reference

If a Snap fails to build due to a proxy error you can use the LaunchPad API to request a rebuild:

1. Start the LaunchPad shell with lp-shell production devel
2. To rebuild arm64 and ppc64el (for example), you would do:

snap = lp.load('/~maas-committers/+snap/maas-2.9')
for arch in ('arm64', 'ppc64el'):
    snap.requestBuild(archive=snap.auto_build_archive, distro_arch_series='/ubuntu/focal/%s' % arch, pocket=snap.auto_build_pocket, channels={"snapcraft": "stable"})