Updated to maas 2.9.2, updated python-libmaas, and now it's broken :(

I updated our maas staging environment to 2.9.2 with snap (moved DB off host to it’s own DB server, rebuilt region and reack controllers as 20.04 and re-paired. Using my home grown tool which leverages python-libmaas to talk to maas and do its thing. Updated python-libmaas to the current tag (0.6.6) Commissioning works great, however deployment does not My code calls machine.restore_networking_configuration() which previously worked against maas 2.6.2 but now fails against 2.9.2 I tried using machine.restore_default_configuration() instead and it results in pretty much the same stack trace.

      File "./maasterblaster.py", line 720, in cleanup_machine
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/utils/maas_async.py", line 43, in wrapper
        result = eventloop.run_until_complete(result)
      File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
        return future.result()
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/viscera/machines.py", line 788, in restore_default_configuration
        await self._handler.restore_default_configuration(system_id=self.system_id)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/bones/__init__.py", line 307, in __call__
        response = await self.bind(**params).call(**data)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/maas/client/bones/__init__.py", line 459, in dispatch
        response = await session.request(
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
        await resp.start(conn)
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 847, in start
        message, payload = await self._protocol.read()  # type: ignore  # noqa
      File "/home/dandruczyk/libmaas/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read
        await self._waiter

This is blocking our production upgrade which is sorely needed because maas 2.6.2 does NOT work well with 1000+ machines in it’s DB. :frowning: Help!

Hi @dandruczyk,

Sorry you’re hitting this issue - we’ve tried a couple of times to reproduce it but are having no luck.

I wonder if you could help us understand a bit more about your setup so that we can dig in - is maasterblaster.py (love the name!) running on a machine with direct access to the MAAS server ( TCP :5240) or is there a proxy/load-balancer/ something else in between?

This is running with direct access to the mass server. I.e. http://:5240/MAAS. No LB yet, though that is planned to increase the resiliency once we vet 2.9.x for use.

The bug I filed on the MaaS bug tracker detailed both the server and client stack traces.

This seems to be a bug in python-twisted.

I bet the maas snap was never rebuilt when the above bug was fixed… Can someone please do that?

Either way this is a showstopper and no-one has or yet triaged my bug posted in a previous comment so I’m stuck as are likely others who use the API in an advanced manner.

1 Like

Glad to see that the bug has been identified and patched in the python-twisted package, thank you for linking

A rebuild of 2.9.2 against the latest updates is now in progress, and will show up first in 2.9/candidate channel when it finishes.

nice catch, @dandruczyk. thank you.

Excellent, though it looks like the build failed.
From: https://launchpadlibrarian.net/528428511/buildlog_snap_ubuntu_focal_amd64_maas-2.9.2_BUILDING.txt.gz

  • snapcraftctl prime
    /bin/bash: line 41: SNAPCRAFT_PART_INSTALL: unbound variable
    Failed to run ‘override-prime’: Exit code was 1.
    Build failed
    Traceback (most recent call last):
    File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 258, in run
    File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 247, in build
    File “/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py”, line 100, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
    File “/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py”, line 537, in run
    subprocess.check_call(cmd, **kwargs)
    File “/usr/lib/python3.6/subprocess.py”, line 311, in check_call
    raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command ‘[‘lxc’, ‘exec’, ‘lp-focal-amd64’, ‘–env’, ‘LANG=C.UTF-8’, ‘–env’, ‘SHELL=/bin/sh’, ‘–env’, ‘SNAPCRAFT_BUILD_INFO=1’, ‘–env’, ‘SNAPCRAFT_IMAGE_INFO={“build-request-id”: “lp-62862874”, “build-request-timestamp”: “2021-03-17T18:06:11Z”, “build_url”: “https://launchpad.net/~maas-committers/+snap/maas-2.9.2/+build/1338209”}’, ‘–env’, ‘SNAPCRAFT_BUILD_ENVIRONMENT=host’, ‘–env’, ‘http_proxy=’, ‘–env’, ‘https_proxy=’, ‘–env’, ‘GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy’, ‘–’, ‘/bin/sh’, ‘-c’, ‘cd /build/maas && linux64 snapcraft’]’ returned non-zero exit status 2.
    Revoking proxy token…
    RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=focal --arch=amd64 SNAPBUILD-1338209
    Scanning for processes to kill in build SNAPBUILD-1338209

tl;dr: update available in 2.9/candidate

There were several issues hit whilst attempting to rebuild 2.9.2 snap

  1. go snap update to 1.16 caused Go module support to be on by default
  2. snapcraft 4.5 change to execute builds with set -u in the shell to complain about undefined variables

The first was worked around to point to go 1.14 channel instead of stable:

--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -237,7 +237,7 @@ parts:
     plugin: make
     source: src/machine-resources
-      - go
+      - go/1.14/stable
       - go-dep

The second was to point to a snapcraft 4.4 branch instead of stable.

With both of these workarounds in place, we have been able to successfully rebuild 2.9.2 with no consequential change and have published that to the snap store in the 2.9/candidate channel.

Both issues are fixed upstream in MAAS master - the first by moving to native go modules, the second by removing the stray variable and additionally been backported to the 2.9 branch for the upcoming 2.9.3 release.

I installed the 2.9.2 snap from the canidate channel on my test environment region and rack controllers an was able to use the API now with my custom code (maasterblaster) successfully.

Thanks everyone!
maas 2.9.2-9165-g.668f61183 12504 2.9/candidate canonical✓ -


This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.