Problem of ssh connection to MAAS VM (permission denied on rescue mode)

Users
, ,
1

Hi all,
I am following this tutorial https://www.youtube.com/watch?v=sLADei_c9Qg, to deploy Kubernetes with MAAS and juju.

Yesterday everything was working fine and I switch off the 3 maas VM and the host server. Today when restarting it I could see that Juju could not start VM 3:

image
image928×167 3.88 KB

So I tried to ssh to the metal-3 but I was unsuccessful, however I managed to ssh to the 2 others Vms:

❯ juju ssh ubuntu@10.10.10.22
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-104-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Mar 16 09:55:42 UTC 2022

  System load:  0.01               Processes:                  246
  Usage of /:   45.0% of 22.30GB   Users logged in:            0
  Memory usage: 12%                IPv4 address for br-enp5s0: 10.10.10.22
  Swap usage:   0%                 IPv4 address for lxdbr0:    10.84.156.1

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

0 updates can be applied immediately.


*** System restart required ***

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@metal-2:~$ exit
logout
Connection to 10.10.10.22 closed.
❯ juju ssh ubuntu@10.10.10.21
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-104-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Mar 16 09:55:50 UTC 2022

  System load:  0.05               Processes:                  259
  Usage of /:   69.5% of 22.30GB   Users logged in:            0
  Memory usage: 15%                IPv4 address for br-enp5s0: 10.10.10.21
  Swap usage:   0%                 IPv4 address for lxdbr0:    10.184.251.1

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

3 updates can be applied immediately.
3 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable


Last login: Wed Mar 16 09:46:09 2022 from 10.10.10.1
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@metal-1:~$ exit
logout
Connection to 10.10.10.21 closed.
❯ juju ssh ubuntu@10.10.10.23
ssh: connect to host 10.10.10.23 port 22: Connection refused

I also tried the rescue mode on 10.10.10.23 . When I do this, I manage to connect by ssh but I got a Permission denied (publickey):

juju ssh ubuntu@10.10.10.23 The authenticity of host '10.10.10.23 (10.10.10.23)' can't be established. ECDSA key fingerprint is SHA256:GrlGZtORryfRApZVxK0bSyFgy88arJ4ftQr6zfuucn8. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '10.10.10.23' (ECDSA) to the list of known hosts. ubuntu@10.10.10.23: Permission denied (publickey).

2

@lhoupert, just saw this one – :eyes:

3

have you tried creating another VM and shutting it off, to see if it behaves similarly – and grabbing some logs while you’re trying to reproduce it? or maybe you have the logs from the machine that failed, around the time it failed?

it looks like something (a public key?) got lost when you shut the machine off, but that’s pure speculation. was your .ssh directory ephemeral in some way, so that it didn’t get preserved across reboots? again, just wildly guessing.

if you can send me some log files, i can take a look.

4

Hi Bill, I could not create more VMs as all my CPU resources were allocated to these 3 VMs.
The .ssh directory was not ephemeral, the three VMs were created the same way, but only one could not be reached after reboot.

If there any best practices for switching off MAAS and the VM associated before switching off the host computer?

I cannot access this setup, I wiped out to start again from a clean ubuntu install. But I now have a problem in the VM creation step (VM host does not support composability although /dev/kvm exists + "Error: An architecture is required" in GUI).

5

@lhoupert, this might not be a bug, but please file one anyway – some of what you’re saying sounds like things i’ve seen in other bugs. at the very least, we can make sure it’s not an actual code issue. thanks.

1 Like
6

@billwear Thank you, I filled a report here https://bugs.launchpad.net/maas/+bug/1970587

I will also fill one for the composability problem as soon as I have a bit more time.

7

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.

8

thanks, @lhoupert. closing this post so the we have one source of truth, here (the bug).

1 Like