MaaS 2.5 / Bionic / Pod, Composing guests = Kernel Panic, no init

Users
1

Hey guys,

I just installed MaaS 2.5 on Ubuntu 18.04 (inside of a LXD Container).

Then, I deployed Ubuntu 18.04 in a bare metal SuperMicro Blade, all good!

After deploying it, I installed manually KVM and added it as a Pod on MaaS, so far, so good…

I manage to Compose one, two guests and then, out of blue, I can not Compose more guests anymore!

Right after trying to Compose, the KVM Guest dies with Kernel Panic, as follows:

https://imgur.com/a/rc08eGQ

I already reinstalled everything (including MaaS) from scratch and I can not find the problem!

What’s happening, any idea?

Thanks!
Thiago

2

BTW, I also tried to manually create a KVM Guest using Virt-Manager, via PXE, same Kernel Panic!

No idea about what to do…

I can release/deploy other bare metal boxes normally!

I can even see on MaaS’ logs, that the machines downloaded the same kernel and initrd from tftp…

So sad… :frowning:

3

I’m starting to think that the boot-kernel/boot-initrd locate inside of my boot-resources are broken!

4

This is blocking me to deploy MaaS 2.5 in production! I thought that it was ready / stable but, looks like that it isn’t.

I was using MaaS 2.5 for months and it was working perfectly! Now I can’t Compose more KVM Guests. For some reason, it works for 1 or 2 guests, then, it stops…

What is happening? :sob:

I would love to debug this problem! I know that KVM can boot kernel/initrd directly but, is it possible to use it to debug this problem? I mean, I think that the boot-resources are currently broken! And those are updated daily, so, might not be directly related to MaaS itself…

5

We are currently tracking this issue in LP:1810972 Is your MAAS running in a virtual environment or on metal? I am currently investigating if this is an issue with nginx’s sendfile. Could you try setting sendfile off; in /var/lib/maas/http/nginx.conf and restarting nginx with systemctl restart maas-http?

6

Ok! I’ll try this as well now.

However, you’ll not believe in my workaround! I “fixed” it in a very cool way… lol

I’ll post about my workaround very soon!

7

This option had no effect. Composed KVM Guest in kernel panic, no init available.

8

My workaround is to build “Compose” the KVM Guest using Virt-Manager AND using UEFI!!!

To do that, I have to run on each KVM Host:

sudo apt install ovmf

Then, compose the KVM Guests using “Firmware = UEFI”, instead of “BIOS”.

Voialá! Problem solved!

No more kernel panic and MaaS can take over the vm after a simple “Pod Refresh”!!!

It’s even better! When deleting a Pod, even accidentally, ALL the VMs will be left untouched! I really like this. So, no more Composing KVM Guests via MaaS UI for me (for now).

9

For the wider audience, the issue in reality is the fact that there’s a kernel panic with 2048 MB of ram allocated to the VM. This is a kernel or a qemu/KVM issue.

For more information, please follow https://bugs.launchpad.net/qemu/+bug/1797581

10