Issue with MaaS CLI after upgrade to 3.5 stable

Users
1

Hello,

It seems like after upgrading to MaaS 3.5 stable, the cli has stopped fully working. The available commands are limited, and running commands result in a HTTP signature error.

[root@util01 ~]# maas apikey --username=syseng > api-key 
[root@util01 ~]# maas login admin http://127.0.0.1:5240/MAAS/api/2.0/ < api-key
API key (leave empty for anonymous access): 

You are now logged in to the MAAS server at
http://127.0.0.1:5240/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help

[root@util01 ~]# maas admin --help
usage: maas admin [-h] COMMAND ...

Issue commands to the MAAS region controller at http://127.0.0.1:5240/MAAS/api/2.0/.

options:
  -h, --help          show this help message and exit

drill down:
  COMMAND
    devices           Manage the collection of all the devices in the MAAS.
    files             Manage the collection of all the files in this MAAS.
    machines          Manage the collection of all the machines in the MAAS.
    nodes             Manage the collection of all the nodes in the MAAS.
    rack-controllers  Manage the collection of all rack controllers in MAAS.
    region-controllers
                      Manage the collection of all region controllers in MAAS.
    version           Information about this MAAS instance.

This is a profile.  Any commands you issue on this profile will
operate on the MAAS region server.

The command information you see here comes from the region server's
API; it may differ for different profiles.  If you believe the API may
have changed, use the command's 'refresh' sub-command to fetch the
latest version of this help information from the server.
[root@util01 ~]# maas admin machines read
Unrecognised signature: method=GET op=None

Looking at logs, I think the http log is the only one with information relating to this.

Aug 14 11:45:40 util01 maas-http[2866400]:  127.0.0.1 - - [14/Aug/2024:11:45:40 -0400] "GET /MAAS/api/2.0/machines/ HTTP/1.1" 400 42 "-" "Python-httplib2/0.20.2 (gzip)"

Happy to do further investigations, any help is welcome.

2

Hi @jcoleman-lw

May I ask you to run maas refresh and try again?
Also I’d be curious about cksum $(which maas) output

3 
[root@util01 ~]# cksum $(which maas)
1088676795 17169664 /var/lib/snapd/snap/bin/maas
[root@util01 ~]# maas refresh
[root@util01 ~]# cksum $(which maas)
1088676795 17169664 /var/lib/snapd/snap/bin/maas
[root@util01 ~]# maas admin machines read
Unrecognised signature: method=GET op=None

Doesn’t seem like any changes were made. Also, as a note, the webui and API calls from the web browser seems to be ok.

4

Hmm, so far I am failing to reproduce this.

Which version of MAAS you had before doing update?
Is there anything in the Region controller logs?

$> journalctl -fu snap.maas.pebble -t maas-regiond

Would be interesting to get output of the following commands:

$> snap list | grep maas
$> maas admin version read --debug
5

Nothing really useful out of the logs for regiond.

[root@utl01 ~]# journalctl -fu snap.maas.pebble -t maas-regiond
Aug 15 10:01:32 utl01 maas-regiond[2900896]: maasserver.ipc: [info] Worker pid:2900936 lost burst connection to ('1.2.3.4', 5250).
Aug 15 10:01:52 utl01 maas-regiond[2900936]: twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=54674, flowInfo=0, scopeID=0))
Aug 15 10:01:52 utl01 maas-regiond[2900936]: twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=54690, flowInfo=0, scopeID=0))
Aug 15 10:01:52 utl01 maas-regiond[2900936]: twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:1.2.3.4', port=54698, flowInfo=0, scopeID=0))
Aug 15 10:01:52 utl01 maas-regiond[2900936]: maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:1.2.3.4:54698'.
Aug 15 10:01:52 utl01 maas-regiond[2900936]: maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:1.2.3.4:54674'.
Aug 15 10:01:52 utl01 maas-regiond[2900936]: maasserver.rpc.regionservice: [info] Rack controller authenticated from '::ffff:1.2.3.4:54690'.
Aug 15 10:01:53 utl01 maas-regiond[2900896]: maasserver.ipc: [info] Worker pid:2900936 registered RPC connection to ('63xqp7', '1.2.3.4', 5250).
Aug 15 10:01:54 utl01 maas-regiond[2900896]: maasserver.ipc: [info] Worker pid:2900936 registered RPC connection to ('63xqp7', '1.2.3.4', 5250).
Aug 15 10:01:54 utl01 maas-regiond[2900896]: maasserver.ipc: [info] Worker pid:2900936 registered RPC connection to ('63xqp7', '1.2.3.4', 5250).
^C
[root@utl01 ~]# snap list | grep maas
maas    3.5.0-16308-g.c799a1080  36363  3.5/stable     canonical**  -
[root@utl01 ~]# maas admin version read --debug
200 OK

        Connection: keep-alive
    Content-Length: 285
  Content-Location: http://127.0.0.1:5240/MAAS/api/2.0/version/
      Content-Type: application/json; charset=utf-8
              Date: Thu, 15 Aug 2024 14:03:04 GMT
            Server: nginx/1.18.0 (Ubuntu)
            Status: 200
              Vary: Authorization
   X-Frame-Options: DENY
   X-Maas-Api-Hash: d4XXXXXXX5c36

Success.
Machine-readable output follows:
{"capabilities": ["networks-management", "static-ipaddresses", "ipv6-deployment-ubuntu", "devices-management", "storage-deployment-ubuntu", "network-deployment-ubuntu", "bridging-interface-ubuntu", "bridging-automatic-ubuntu", "authenticate-api"], "version": "3.5.0", "subversion": ""}
6

Oh yeah, the version we had before was MaaS 3.4 stable.

7

I will give it a try in couple hours and get back to you with results.

8

Hm, nope, I cannot reproduce it (I migrated from 3.4.4 → 3.5.0)

I can only suggest to try to nuke ~/snap/maas/current/.maascli.db (you will lose all your existing sessions) and then call maas login again

ubuntu@maas-tester:~$ sudo snap install maas-test-db --channel=3.5/stable && sudo snap install maas --channel=3.4/stable
maas-test-db (3.5/stable) 14.2-34-g.f09c893 from Canonical✓ installed
maas (3.4/stable) 3.4.4-14367-g.8f9c76b72 from Canonical✓ installed

ubuntu@maas-tester:~$ sudo maas init region+rack
Database URI [default=maas-test-db:///]:
MAAS URL [default=http://10.166.116.58:5240/MAAS]:
MAAS has been set up.

If you want to configure external authentication or use
MAAS with Canonical RBAC, please run

  sudo maas configauth

To create admins when not using external authentication, run

  sudo maas createadmin

To enable TLS for secured communication, please run

  sudo maas config-tls enable

ubuntu@maas-tester:~$ sudo maas createadmin --username admin --password admin --email admin@localhost --ssh-import gh:troyanov

ubuntu@maas-tester:~$ sudo maas apikey --username admin
EQ5FzVRa6xTJ7rTqAn:BR6bV7GLUx69apZKVn:7VA43nnxK7Pv7ZkpfsfmL3PMsr9h5wmu

ubuntu@maas-tester:~$ maas login admin http://localhost:5240/MAAS EQ5FzVRa6xTJ7rTqAn:BR6bV7GLUx69apZKVn:7VA43nnxK7Pv7ZkpfsfmL3PMsr9h5wmu

You are now logged in to the MAAS server at
http://localhost:5240/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help


ubuntu@maas-tester:~$ maas admin version read
Success.
Machine-readable output follows:
{"capabilities": ["networks-management", "static-ipaddresses", "ipv6-deployment-ubuntu", "devices-management", "storage-deployment-ubuntu", "network-deployment-ubuntu", "bridging-interface-ubuntu", "bridging-automatic-ubuntu", "authenticate-api"], "version": "3.4.4", "subversion": ""}

ubuntu@maas-tester:~$ maas admin machines read
Success.
Machine-readable output follows:
[]

ubuntu@maas-tester:~$ sudo snap refresh maas --channel=3.5/stable
2024-08-15T18:24:51Z INFO Waiting for "snap.maas.supervisor.service" to stop.
maas (3.5/stable) 3.5.0-16308-g.c799a1080 from Canonical✓ refreshed

ubuntu@maas-tester:~$ maas admin version read
**********************************************************************
*** WARNING! The API on the server differs from the description that
*** is cached locally. This may result in failed API calls. Refresh
*** the local API description with `maas refresh`.
**********************************************************************
Success.
Machine-readable output follows:
{"capabilities": ["networks-management", "static-ipaddresses", "ipv6-deployment-ubuntu", "devices-management", "storage-deployment-ubuntu", "network-deployment-ubuntu", "bridging-interface-ubuntu", "bridging-automatic-ubuntu", "authenticate-api"], "version": "3.5.0", "subversion": ""}

ubuntu@maas-tester:~$ maas refresh

ubuntu@maas-tester:~$ maas admin version read
Success.
Machine-readable output follows:
{"capabilities": ["networks-management", "static-ipaddresses", "ipv6-deployment-ubuntu", "devices-management", "storage-deployment-ubuntu", "network-deployment-ubuntu", "bridging-interface-ubuntu", "bridging-automatic-ubuntu", "authenticate-api"], "version": "3.5.0", "subversion": ""}

ubuntu@maas-tester:~$ maas admin machines read
Success.
Machine-readable output follows:
[]
9

Thanks for your assistance, however it did not seem to work. I do have some developer experience, maybe I can add some debug to output how its performing the authentication and the http request its making?

[root@utl01 ~]# rm -Rf snap/
[root@utl01 ~]# rm /var/snap/maas/36280/root/snap/maas/34087/.maascli.db
rm: remove regular file '/var/snap/maas/36280/root/snap/maas/34087/.maascli.db'? y
[root@utl01 ~]# rm /var/snap/maas/36280/root/snap/maas/36280/.maascli.db
rm: remove regular file '/var/snap/maas/36280/root/snap/maas/36280/.maascli.db'? y
[root@utl01 ~]# rm /var/snap/maas/36363/root/snap/maas/34087/.maascli.db
rm: remove regular file '/var/snap/maas/36363/root/snap/maas/34087/.maascli.db'? y
[root@utl01 ~]# rm /var/snap/maas/36363/root/snap/maas/36280/.maascli.db
rm: remove regular file '/var/snap/maas/36363/root/snap/maas/36280/.maascli.db'? y
[root@utl01 ~]# rm /var/snap/maas/36363/root/snap/maas/36363/.maascli.db
rm: remove regular file '/var/snap/maas/36363/root/snap/maas/36363/.maascli.db'? y
[root@utl01 ~]# updatedb
[root@utl01 ~]# locate maascli.db
[root@utl01 ~]# maas login admin http://127.0.0.1:5240/MAAS/api/2.0/ < api-key
API key (leave empty for anonymous access): 

You are now logged in to the MAAS server at
http://127.0.0.1:5240/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help

[root@utl01 ~]# maas admin machines read
Unrecognised signature: method=GET op=None
[root@utl01 ~]# ls -lah snap/maas/
total 0
drwxr-xr-x. 4 root root 48 Aug 16 12:58 .
drwx------. 3 root root 18 Aug 16 12:58 ..
drwxr-xr-x. 2 root root  6 Aug 16 12:58 36363
drwxr-xr-x. 2 root root  6 Aug 16 12:58 common
lrwxrwxrwx. 1 root root  5 Aug 16 12:58 current -> 36363
[root@utl01 ~]# ls -lah snap/maas/current/
total 0
drwxr-xr-x. 2 root root  6 Aug 16 12:58 .
drwxr-xr-x. 4 root root 48 Aug 16 12:58 ..
[root@utl01 ~]# updatedb
[root@utl01 ~]# locate maascli.db
/var/snap/maas/36363/root/snap/maas/36363/.maascli.db
10

I guess the best way to track down the issue would be capturing traffic with tcpdump

11

I just did a fresh install of 3.5 stable, after having tested 3.4 for a a few months. In comparison, I am noticing extremely slugglish and slow maas cli responses. Some don’t even finish. Upon exiting a given cmd (crtl-c) I see this as an example: sudo maas createadmin
^CTraceback (most recent call last):
File “/snap/maas/36889/bin/maas”, line 5, in
from maascli import main
File “/snap/maas/36889/lib/python3.10/site-packages/maascli/init.py”, line 10, in
from maascli.parser import get_deepest_subparser, prepare_parser
File “/snap/maas/36889/lib/python3.10/site-packages/maascli/parser.py”, line 12, in
from maascli.cli import register_cli_commands
File “/snap/maas/36889/lib/python3.10/site-packages/maascli/cli.py”, line 18, in
from maascli.auth import (
File “/snap/maas/36889/lib/python3.10/site-packages/maascli/auth.py”, line 12, in
from macaroonbakery import httpbakery
File “/snap/maas/36889/usr/lib/python3/dist-packages/macaroonbakery/httpbakery/init.py”, line 3, in
from ._client import (
File “/snap/maas/36889/usr/lib/python3/dist-packages/macaroonbakery/httpbakery/_client.py”, line 7, in
import macaroonbakery.bakery as bakery
File “/snap/maas/36889/usr/lib/python3/dist-packages/macaroonbakery/bakery/init.py”, line 75, in
from ._oven import (
File “/snap/maas/36889/usr/lib/python3/dist-packages/macaroonbakery/bakery/_oven.py”, line 29, in
from ._internal import id_pb2
File “/snap/maas/36889/usr/lib/python3/dist-packages/macaroonbakery/bakery/_internal/id_pb2.py”, line 6, in
from google.protobuf import descriptor as _descriptor
File “/snap/maas/36889/usr/lib/python3/dist-packages/google/protobuf/descriptor.py”, line 40, in
from google.protobuf.internal import api_implementation
File “/snap/maas/36889/usr/lib/python3/dist-packages/google/protobuf/internal/api_implementation.py”, line 104, in
from google.protobuf.pyext import _message
File “/usr/lib/python3.10/abc.py”, line 110, in register
def register(cls, subclass):
KeyboardInterrupt

12

It would be useful to collect some numbers to understand if there is a regression or if it’s just “a feeling”

13

Adding more RAM really helped the issue. Seems to be fine now. Running 3GB for a single instance test controller.