Machines deploying as read-only file systems

Hi all,

Bit of a weird one here that’s got us scratching our heads, so I thought I’d ask here in case someone has come across this before.

I’ll outline our scenario first, and then the problem.

We are using packer to build our MAAS images, using docker as the builder.

For the docker image, we are using the bionic cloud image as a source, and converted into a docker image. This works. If I import this image into MAAS, and deploy a machine, it works as expected. Deployment completes, cloud-init runs etc.

Now the odd bit… when we go to run this through packer to add our own customisations, it fails. Even with a very simple packer script (included below, for reference - you can see it’s not doing anything specific), the machine will get stuck at deploying, and if you boot into recovery mode, the file system is stuck as read-only. You can remount it as rw, but every reboot will make it ro again.

This has been puzzling us for a while, so does anyone have anything that they can point us too, please?!

Thanks!
Joe

{
    "builders": [
        {
            "type": "docker",
            "image": "our-repo/ubuntu-bionic-cloud-image:v18.04.6",
            "privileged": "true",
            "export_path": "output/cloud-image-no-changes.tar"
        }
    ],
    "post-processors": [
        {
            "type": "compress",
            "output": "output/cloud-image-no-changes.gz",
            "compression_level": 1
        }
    ]
}

Hi @joealford,

what are the steps you used to create the image that works? Is it a raw (dd.gz) or a tar.gz image?

Hi @alexsander-souza,

TL:DR - it’s a tar.gz image.

To create the working MAAS image, I took the URL above and did the following:

xz -dv bionic-server-cloudimg-amd64-root.tar.xz
gzip -v bionic-server-cloudimg-amd64-root.tar

My guess is that Docker is messing with the /dev contents, but to be certain you need to unpack the images (be careful to preserve special files), run md5sum in both and then compare the results.

Thanks - I looked into /dev this morning, but that didn’t solve it. It was nothing to do with read-only file system in the end. I’d been going around in circles for so long, I think I’d gone slightly mad and forgot if you boot into recovery mode it’s always a ro file system :man_facepalming:

I can happily report that it is fixed though. As we were using docker to build our images, and docker mounts /etc/resolv.conf externally, we just had an empty /etc/resolv.conf file, instead of a symlink to /run/systemd/resolve/stub-resolv.conf. Once I’d added some commands to packer to fix this, MAAS was able to deploy the image and cloud-init was able to run.

In case it helps anyone in the future, this was the magic needed.

    systemd_stub_dir="/run/systemd/resolve/"
    systemd_stub_file="stub-resolv.conf"
    resolv_conf="/etc/resolv.conf"

    mkdir -p $systemd_stub_dir
    touch $systemd_stub_dir$systemd_stub_file
    umount $resolv_conf #to perform this, we have to run as a privileged container.
    rm -f $resolv_conf
    ln -s $systemd_stub_dir$systemd_stub_file $resolv_conf
1 Like

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.