PXE-E16 no valid offer received

hypeitnow · 5 March 2023 07:51

Hi all,

I just set up my standard MAAS ha installation, and have configured the pxe booting in my VM(adapter has only ipv4, no dhcp), maas knows that network and is able to see ip range.
I see following one which I do not understand:

I did some testing which exclude networking issues

root@u1:~# ping google.com
PING google.com (142.250.75.14) 56(84) bytes of data.
64 bytes from waw07s03-in-f14.1e100.net (142.250.75.14): icmp_seq=1 ttl=116 time=46.1 ms
64 bytes from waw07s03-in-f14.1e100.net (142.250.75.14): icmp_seq=2 ttl=116 time=46.1 ms
64 bytes from waw07s03-in-f14.1e100.net (142.250.75.14): icmp_seq=3 ttl=116 time=46.0 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 45.952/46.058/46.132/0.077 ms
root@u1:~# apt install telnet
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  telnet
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 66.9 kB of archives.
After this operation, 158 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 telnet amd64 0.17-44build1 [66.9 kB]
Fetched 66.9 kB in 0s (325 kB/s)
Selecting previously unselected package telnet.
(Reading database ... 21754 files and directories currently installed.)
Preparing to unpack .../telnet_0.17-44build1_amd64.deb ...
Unpacking telnet (0.17-44build1) ...
Setting up telnet (0.17-44build1) ...
update-alternatives: using /usr/bin/telnet.netkit to provide /usr/bin/telnet (telnet) in auto mode
root@u1:~# telnet 10.10.99.101 5443
Trying 10.10.99.101...
Connected to 10.10.99.101.
Escape character is '^]'.
^CConnection closed by foreign host.
root@u1:~# telnet 10.10.99.101 5248
Trying 10.10.99.101...
Connected to 10.10.99.101.
Escape character is '^]'.
^CConnection closed by foreign host.
root@u1:~# wget
-bash: wget: command not found
root@u1:~# apt install wget
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  wget
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 367 kB of archives.
After this operation, 1,008 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 wget amd64 1.21.2-2ubuntu1 [367 kB]
Fetched 367 kB in 1s (461 kB/s)
Selecting previously unselected package wget.
(Reading database ... 21763 files and directories currently installed.)
Preparing to unpack .../wget_1.21.2-2ubuntu1_amd64.deb ...
Unpacking wget (1.21.2-2ubuntu1) ...
Setting up wget (1.21.2-2ubuntu1) ...
root@u1:~# wget http://10.10.99.101:5248/images/boot64.efi
--2023-03-05 07:38:04--  http://10.10.99.101:5248/images/boot64.efi
Connecting to 10.10.99.101:5248... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-05 07:38:04 ERROR 404: Not Found.

root@u1:~# wget http://10.10.99.101:5248/images/bootx64.efi
--2023-03-05 07:38:42--  http://10.10.99.101:5248/images/bootx64.efi
Connecting to 10.10.99.101:5248... connected.
HTTP request sent, awaiting response... 200 OK
Length: 955656 (933K) [application/octet-stream]
Saving to: ‘bootx64.efi’

bootx64.efi                                    100%[====================================================================================================>] 933.26K  --.-KB/s    in 0.05s

2023-03-05 07:38:42 (18.7 MB/s) - ‘bootx64.efi’ saved [955656/955656]

I also tried to set up another lxd bridge as in https://www.youtube.com/watch?v=b_bdVfG47G4
but the effect is exactly the same…

My setup is maas 3.4 and postgres 15.
@troyanov @billwear the documentation does not mention particular postgres version but I assume that the newest would suite it.

How can I troubleshoot it further? Are there some particular seetings in maas I need to pay attention to?

Kind regards

Mateusz

hypeitnow · 5 March 2023 13:28

I found the problem that is causing the issue, the tftp not working is caused by rackd outage which is caused by setting up https address in rackd.conf.

I recently opened an issue asking about the tls in maas, what I did not mention though is that all my 3 maas servers are initialised in rack plus region mode.

Now when the address in rackd.conf became https://10.10.10.104 I keep getting the following error

2023-03-05 12:47:59 provisioningserver.rpc.clusterservice: [critical] Failed to contact region. (While requesting RPC info at https://10.10.10.104/MAAS).
        Traceback (most recent call last):
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 661, in callback
            self._startRunCallbacks(result)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
            current_context.run(_inlineCallbacks, r, gen, status)
        --- <exception caught here> ---
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1299, in _doUpdate
            eventloops, maas_url = yield self._get_rpc_info(urls)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1558, in _get_rpc_info
            raise config_exc
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1529, in _get_rpc_info
            eventloops, maas_url = yield self._parallel_fetch_rpc_info(urls)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1503, in handle_responses
            errors[0].raiseException()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1464, in _serial_fetch_rpc_info
            raise last_exc
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1456, in _serial_fetch_rpc_info
            response = yield self._fetch_rpc_info(url, orig_url)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
            result = current_context.run(
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1558, in _get_rpc_info
            raise config_exc
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1529, in _get_rpc_info
            eventloops, maas_url = yield self._parallel_fetch_rpc_info(urls)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1503, in handle_responses
            errors[0].raiseException()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
            result = current_context.run(
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1464, in _serial_fetch_rpc_info
            raise last_exc
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/rpc/clusterservice.py", line 1456, in _serial_fetch_rpc_info
            response = yield self._fetch_rpc_info(url, orig_url)
        twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'certificate verify failed')]>]

DNS is also failing to start properly, most likely to rack controllers no having started properly:

2023-03-05 12:46:25 maasserver.ipc: [info] Worker pid:805 registered RPC connection to ('pket84', '10.10.99.101', 5251).
2023-03-05 12:46:26 maasserver.region_controller: [critical] Failed configuring DNS; killing and restarting
        Traceback (most recent call last):
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
            return _cancellableInlineCallbacks(gen)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
            _inlineCallbacks(None, gen, status)
        --- <exception caught here> ---
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/region_controller.py", line 287, in _checkSerial
            raise DNSReloadError(
        maasserver.region_controller.DNSReloadError: Failed to reload DNS; timeout or rdnc command failed.

2023-03-05 12:46:26 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'ReportNeighbours' command. [maas-ha-1:pid=804:cmd=ReportNeighbours:ask=b3]
        Traceback (most recent call last):
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
            self.runUntilCurrent()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1138, in checkKnownErrors
            key = error.trap(*command.allErrors)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork()  # type: ignore[attr-defined]
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
            return func(*args, **kwargs)
          File "/snap/maas/26421/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/utils/orm.py", line 771, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/utils/orm.py", line 587, in retrier
            return func(*args, **kwargs)
          File "/usr/lib/python3.10/contextlib.py", line 79, in inner
            return func(*args, **kwds)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/rpc/rackcontrollers.py", line 276, in report_neighbours
            rack_controller.report_neighbours(neighbours)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/models/node.py", line 6448, in report_neighbours
            interface.update_neighbour(
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/models/interface.py", line 1567, in update_neighbour
            neighbour.save(update_fields=["time", "count", "updated"])
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/models/cleansave.py", line 48, in save
            return super().save(*args, **kwargs)
          File "/snap/maas/26421/lib/python3.10/site-packages/maasserver/models/timestampedmodel.py", line 78, in save
            return super().save(*args, **kwargs)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/base.py", line 739, in save
            self.save_base(using=using, force_insert=force_insert,
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/base.py", line 776, in save_base
            updated = self._save_table(
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/base.py", line 858, in _save_table
            updated = self._do_update(base_qs, using, pk_val, values, update_fields,
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/base.py", line 912, in _do_update
            return filtered._update(values) > 0
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/query.py", line 802, in _update
            return query.get_compiler(self.db).execute_sql(CURSOR)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
            cursor = super().execute_sql(result_type)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
            cursor.execute(sql, params)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 66, in execute
            return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
            return executor(sql, params, many, context)
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 79, in _execute
            with self.db.wrap_database_errors:
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/utils.py", line 90, in __exit__
            raise dj_exc_value.with_traceback(traceback) from exc_value
          File "/snap/maas/26421/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
            return self.cursor.execute(sql, params)
        django.db.utils.OperationalError: could not serialize access due to concurrent update

I know that setting it up back to http will not resolve this as nginx will do automatic tls termination.

Is there any reasonable way to approach this?

troyanov · 7 March 2023 12:10

Hi @hypeitnow

Rack controller should use HTTP (now HTTPS) in the config. We currently do have exceptions for some of the API endpoints which are available over 5240 (non-HTTPS)

hypeitnow · 18 March 2023 13:08

Hi @troyanov,

I managed to resolve the issue with rackd and regiond by changing the certificates(added IP of all my machines to SAN of my certificate) and by reconfiguring the addresses in rackd.conf and regiond.conf it seems to work now, but the issu still persists, tftp is turned on so maas should offer the boot img:

MAAS sees the network in which the vm should PXE boot (99)

In maas the DHCP for that network is also configured and the network does not have conflicting dhcp which could cause the issue:

How can I investigate that further?

Kind regards

Mateusz

troyanov · 22 March 2023 14:51

Hi @hypeitnow

PostgreSQL 15 should be fine to use.

So you still have 500 Error when machine is trying to PXE boot?

Regarding your HA setup with TLS, can you please tell a bit more about it? Is there anything else in front of NGINX (that is bundled with MAAS)?

If you take a look at NGINX region template there are certain URLs that still have to be accessed over plain HTTP (and rack <-> region RPC is among them)

system · 21 March 2024 14:51

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.