Over the past 24 hours or so, I’ve started seeing a lot of deploy failures. I’m
using the maas Python API, and the failures do not appear to be specific to any
particular machine.
CLIENT
The client script uses python3-libmaas version 0.6.4-0ubuntu1 and is essentially this:
self.__client = maas.client.connect(MAAS_BASE_URI, apikey=self.config['apikey'])
machines = self.__client.machines.list()
# Some filtering of machines list occurs, then:
for machine in machines:
logger.info('Asking MAAS to deploy %s', machine.fqdn)
machine.deploy()
This tool has been working pretty reliably for the past year or two, and has not changed in months.
The failures look like this on the client side:
INFO egd: Asking MAAS to deploy vm-008-4269.maas.eng.exagrid.com
POST https://maas.eng.exagrid.com/MAAS/api/2.0/machines/hsnt44/?op=deploy -> HTTP 400 Bad Request (Failed to render preseed: Unable to find MAAS ser…)
SERVER
# snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.1.0-10901-g.f1f8f1505 19835 3.1/stable canonical✓ -
On the server, if I do tail -F /var/snap/maas/common/log/*.log
, I see stuff like this:
==> /var/snap/maas/common/log/maas.log <==
2022-06-14T16:35:46.247632+00:00 eng1713 maas.api: [info] Request from user gitlab-runner to acquire machine: vm-008-4269.maas.eng.exagrid.com (hsnt44)
2022-06-14T16:35:46.282865+00:00 eng1713 maas.node: [info] vm-008-4269: Status transition from READY to ALLOCATED
2022-06-14T16:35:46.295518+00:00 eng1713 maas.node: [info] vm-008-4269: allocated to user gitlab-runner
==> /var/snap/maas/common/log/regiond.log <==
2022-06-14 16:35:46 maasserver.preseed: [warn] WARNING: '/snap/maas/19835/etc/maas/preseeds/curtin_userdata' contains deprecated preseed variables. Please remove: main_archive_directory, ports_archive_directory
2022-06-14 16:35:47 regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/machines/hsnt44/?op=deploy HTTP/1.1 --> 400 BAD_REQUEST (referrer: -; agent: Python/3.8 aiohttp/3.6.2)
Can I turn up tracing somewhere to get more details about why maas thinks these are bad requests?