Updated to maas 2.9.2, updated python-libmaas, and now it's broken :(

dandruczyk · 6 March 2021 00:03

I updated our maas staging environment to 2.9.2 with snap (moved DB off host to it’s own DB server, rebuilt region and reack controllers as 20.04 and re-paired. Using my home grown tool which leverages python-libmaas to talk to maas and do its thing. Updated python-libmaas to the current tag (0.6.6) Commissioning works great, however deployment does not My code calls machine.restore_networking_configuration() which previously worked against maas 2.6.2 but now fails against 2.9.2 I tried using machine.restore_default_configuration() instead and it results in pretty much the same stack trace.

      File "./maasterblaster.py", line 720, in cleanup_machine
        machine.restore_default_configuration()
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/utils/maas_async.py", line 43, in wrapper
        result = eventloop.run_until_complete(result)
      File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
        return future.result()
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/viscera/machines.py", line 788, in restore_default_configuration
        await self._handler.restore_default_configuration(system_id=self.system_id)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/bones/__init__.py", line 307, in __call__
        response = await self.bind(**params).call(**data)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/bones/__init__.py", line 459, in dispatch
        response = await session.request(
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
        await resp.start(conn)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 847, in start
        message, payload = await self._protocol.read()  # type: ignore  # noqa
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read
        await self._waiter
    aiohttp.client_exceptions.ServerDisconnectedError

This is blocking our production upgrade which is sorely needed because maas 2.6.2 does NOT work well with 1000+ machines in it’s DB. Help!

sparkiegeek · 10 March 2021 14:49

Hi @dandruczyk,

Sorry you’re hitting this issue - we’ve tried a couple of times to reproduce it but are having no luck.

I wonder if you could help us understand a bit more about your setup so that we can dig in - is maasterblaster.py (love the name!) running on a machine with direct access to the MAAS server ( TCP :5240) or is there a proxy/load-balancer/ something else in between?

dandruczyk · 11 March 2021 03:52

This is running with direct access to the mass server. I.e. http://:5240/MAAS. No LB yet, though that is planned to increase the resiliency once we vet 2.9.x for use.

The bug I filed on the MaaS bug tracker detailed both the server and client stack traces.

dandruczyk · 16 March 2021 20:52

This seems to be a bug in python-twisted.

I bet the maas snap was never rebuilt when the above bug was fixed… Can someone please do that?

Either way this is a showstopper and no-one has or yet triaged my bug posted in a previous comment so I’m stuck as are likely others who use the API in an advanced manner.

sparkiegeek · 17 March 2021 16:54

Glad to see that the bug has been identified and patched in the python-twisted package, thank you for linking

A rebuild of 2.9.2 against the latest updates is now in progress, and will show up first in 2.9/candidate channel when it finishes.

billwear · 17 March 2021 17:27

nice catch, @dandruczyk. thank you.

dandruczyk · 17 March 2021 21:15

Excellent, though it looks like the build failed.
From: https://launchpadlibrarian.net/528428511/buildlog_snap_ubuntu_focal_amd64_maas-2.9.2_BUILDING.txt.gz

snapcraftctl prime
/bin/bash: line 41: SNAPCRAFT_PART_INSTALL: unbound variable
Failed to run ‘override-prime’: Exit code was 1.
Build failed
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 258, in run
self.build()
File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 247, in build
env=env)
File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 100, in run_build_command
return self.backend.run(args, env=full_env, **kwargs)
File “/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py”, line 537, in run
subprocess.check_call(cmd, **kwargs)
File “/usr/lib/python3.6/subprocess.py”, line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘lxc’, ‘exec’, ‘lp-focal-amd64’, ‘–env’, ‘LANG=C.UTF-8’, ‘–env’, ‘SHELL=/bin/sh’, ‘–env’, ‘SNAPCRAFT_BUILD_INFO=1’, ‘–env’, ‘SNAPCRAFT_IMAGE_INFO={“build-request-id”: “lp-62862874”, “build-request-timestamp”: “2021-03-17T18:06:11Z”, “build_url”: “https://launchpad.net/~maas-committers/+snap/maas-2.9.2/+build/1338209”}’, ‘–env’, ‘SNAPCRAFT_BUILD_ENVIRONMENT=host’, ‘–env’, ‘http_proxy=http://10.10.10.1:8222/’, ‘–env’, ‘https_proxy=http://10.10.10.1:8222/’, ‘–env’, ‘GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy’, ‘–’, ‘/bin/sh’, ‘-c’, ‘cd /build/maas && linux64 snapcraft’]’ returned non-zero exit status 2.
Revoking proxy token…
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=focal --arch=amd64 SNAPBUILD-1338209
Scanning for processes to kill in build SNAPBUILD-1338209

sparkiegeek · 18 March 2021 12:11

tl;dr: update available in 2.9/candidate

There were several issues hit whilst attempting to rebuild 2.9.2 snap

go snap update to 1.16 caused Go module support to be on by default
snapcraft 4.5 change to execute builds with set -u in the shell to complain about undefined variables

The first was worked around to point to go 1.14 channel instead of stable:

--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -237,7 +237,7 @@ parts:
     plugin: make
     source: src/machine-resources
     build-snaps:
-      - go
+      - go/1.14/stable
     build-packages:
       - go-dep

The second was to point to a snapcraft 4.4 branch instead of stable.

With both of these workarounds in place, we have been able to successfully rebuild 2.9.2 with no consequential change and have published that to the snap store in the 2.9/candidate channel.

Both issues are fixed upstream in MAAS master - the first by moving to native go modules, the second by removing the stray variable and additionally been backported to the 2.9 branch for the upcoming 2.9.3 release.

dandruczyk · 18 March 2021 23:28

I installed the 2.9.2 snap from the canidate channel on my test environment region and rack controllers an was able to use the API now with my custom code (maasterblaster) successfully.

Thanks everyone!
maas 2.9.2-9165-g.668f61183 12504 2.9/candidate canonical✓ -

system · 20 March 2021 23:28

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.