Deploy over API fails to render preseed

Users
,
1

Hi,

I’m working on some python scripts for integration similar to spinning up a machine from the pool, running a job and then releasing it when done.

It seems deployment fails for my machines with "Failed to render preseed: " if I don’t push them through a test cycle first though. I was wondering if my code was wrong, so I trying maas local machine deploy --debug, and I’m getting the same error.

nafallo@maas:~$ maas local machine deploy bcdywy --debug 
400 Bad Request

   Content-Length: 26
     Content-Type: text/plain; charset=utf-8
             Date: Wed, 15 Jun 2022 08:06:00 GMT
           Server: TwistedWeb/18.9.0
           Status: 400
             Vary: Cookie
  X-Frame-Options: SAMEORIGIN

Failed to render preseed:

I played around with replacing /etc/maas/preseeds/curtin_userdata with curtin_userdata_custom, but that seems to have been a red herring. It was more likely that I pushed the machine that worked through a test cycle, than modifying the file actually helping.

Any suggestions as to finding the reason for this failed rendering? Logs haven’t helped either.

==> /var/log/maas/maas.log <==
2022-06-15T08:06:00.377574+00:00 maas maas.api: [info] Request from user nafallo to acquire machine: badger.magicalforest.net (bcdywy)
2022-06-15T08:06:00.424662+00:00 maas maas.node: [info] badger: Status transition from READY to ALLOCATED
2022-06-15T08:06:00.443657+00:00 maas maas.node: [info] badger: allocated to user nafallo

==> /var/log/maas/regiond.log <==
2022-06-15 08:06:01 maasserver.preseed: [warn] WARNING: '/etc/maas/preseeds/curtin_userdata' contains deprecated preseed variables. Please remove: main_archive_directory, ports_archive_directory
2022-06-15 08:06:01 regiond: [info] 127.0.0.1 POST /MAAS/api/2.0/machines/bcdywy/?op=deploy HTTP/1.1 --> 400 BAD_REQUEST (referrer: -; agent: Python-httplib2/0.14.0 (gzip))

Cheers,
Nafallo

2

Could be worth mentioning perhaps… I’m using packaged MAAS 3.1 from the upstream PPA.

3

@nafallo, you might glance at this article and see what additional info might help. if you could give us a step-by-step reproducer, that might help a little. but yeah, this is weird.

4

Sure, here is some Python code to test the issue.
Remember to replace the variables with your own.

from oauthlib.oauth1 import SIGNATURE_PLAINTEXT
from requests_oauthlib import OAuth1Session
from requests.exceptions import HTTPError
from time import sleep

api_key = "<your apikey>"
maas_url = "<http://maas.example.org:5240/MAAS>"
system_id = "<your system_id>"

consumer_key, consumer_token, secret = api_key.split(":")
maas = OAuth1Session(consumer_key, resource_owner_key=consumer_token, resource_owner_secret=secret, signature_method=SIGNATURE_PLAINTEXT)

while(True):
  deploy_failed = False
  print("Deploying System")
  deploy = maas.post(f"{maas_url}/api/2.0/machines/{system_id}/", params={"op": "deploy"})
  try:
    deploy.raise_for_status()
  except HTTPError:
    deploy_failed = True
  if deploy_failed:
    print(f"Deploy failed with: {deploy.content.decode()}\nRunning Test\n")
    test = maas.post(f"{maas_url}/api/2.0/machines/{system_id}/", params={"op": "test"})
    while(test.json()["status"] == 21):
      test = maas.get(f"{maas_url}/api/2.0/machines/{system_id}/")
      sleep(20)
  else:
    while(deploy.json()["status"] == 9):
      deploy = maas.get(f"{maas_url}/api/2.0/machines/{system_id}/")
      sleep(20)
    print("Machine Deployed Successfully, Releasing...\n")
    ready = maas.post(f"{maas_url}/api/2.0/machines/{system_id}/", params={"op": "release"})
    while(ready.json()["status"] != 4):
      ready = maas.get(f"{maas_url}/api/2.0/machines/{system_id}/")
      sleep(20)

Sample Output:

Deploying System
Machine Deployed Successfully, Releasing...

Deploying System
Deploy failed with: Failed to render preseed: 
Running Test

Deploying System
Machine Deployed Successfully, Releasing...

Deploying only seems to be working if I run tests in between deploys.

5

Alternative version of the while-loop without automatic testing:

while True:
    read = maas.get(f"{maas_url}/api/2.0/machines/{system_id}/")
    status = read.json()["status"]
    if status == 4:
        print("System Ready, Deploying...")
        deploy = maas.post(
            f"{maas_url}/api/2.0/machines/{system_id}/", params={"op": "deploy"}
        )
        try:
            deploy.raise_for_status()
        except HTTPError:
            print(f"Deploy Failed: {deploy.content.decode()}\n")
    elif status == 6:
        print("Machine Deployed, Releasing...\n")
        maas.post(
            f"{maas_url}/api/2.0/machines/{system_id}/", params={"op": "release"}
        )
    sleep(20)
6

thanks, @nafallo. nice reproducer. we’ll look at this.

1 Like
7

i’ve asked someone to have a look, but they’re in CET, so it might be a day or so before i can respond.

1 Like
8

Hi @nafallo, does the maas local machine deploy bcdywy --debug output contain any additional info after the “failed to render” message?
There should be output from the actual error that’s happening

9

Hi @ack,

It does not, no. What I posted in the initial message is what I get.

10

Same behaviour on 3.2 fwiw. Need to run a test inbetween maas local machine deploy retries to not get the empty Failed to render preseed: message.

11

Just noticed this is affecting juju as well. Not sure I’m surprised.

12

@nafallo, we’re not getting anywhere with this one. have you learned anything new? if not, please go ahead and file a bug and place it against both MAS and curtin. if you can cross-post the bug number here, that’s great, but not essential.

13

I recently hit the issue which had the same symptoms as this. In my case disabling allow_dns on one of the subnets attached to the rack controller and the guest fixed the issue for me. I have raised two bugs to track the issue: