Error: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

vasartori · 18 July 2024 13:27

When we access the “machines” interface, an error is raised: “(‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))”.

This is a new installation of MAAS 3.5.

Looking at logs, we get this stack trace:

Jul 18 13:17:31 maas-regiond01 regiond[7023]:         Traceback (most recent call last):
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3.10/threading.py", line 953, in run
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             self._target(*self._args, **self._kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 821, in worker
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return target()
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 47, in work
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             task()
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 182, in doWork
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             task()
Jul 18 13:17:31 maas-regiond01 regiond[7023]:         --- <exception caught here> ---
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             result = inContext.theWork()  # type: ignore[attr-defined]
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return self.currentContext().callWithContext(ctx, func, *args, **kw)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return func(*args, **kw)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 856, in callInContext
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return func(*args, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             result = func(*args, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 771, in call_within_transaction
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return func_outside_txn(*args, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 574, in retrier
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return func(*args, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3.10/contextlib.py", line 79, in inner
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return func(*args, **kwds)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 431, in prep_user_execute
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return self._call_method_track_queries(
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/websockets/base.py", line 477, in _call_method_track_queries
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             result = method(params)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 359, in list
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             res = self._list_sqlalchemy(params)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 356, in _list_sqlalchemy
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return self.api_client.post("machines", json=res).json()
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasapiserver/client.py", line 32, in post
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return self.request("POST", path, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/maasapiserver/client.py", line 26, in request
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             return self.session.request(verb, url, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             resp = self.send(prep, **send_kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             r = adapter.send(request, **kwargs)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:           File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
Jul 18 13:17:31 maas-regiond01 regiond[7023]:             raise ConnectionError(err, request=request)
Jul 18 13:17:31 maas-regiond01 regiond[7023]:         requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Which file is not found? The error message is not clear.

r00ta · 18 July 2024 14:19

can you extract the apiserver logs with

journalctl -t maas-apiserver

?

vasartori · 18 July 2024 14:23

Huumm, there are no entries in both machines (2)

root@maas-regiond01:~# journalctl -t maas-apiserver 
-- No entries --
root@maas-regiond01:~#

troyanov · 18 July 2024 16:01

@vasartori could it be that you’ve partially updated MAAS to 3.5 and hence some dependencies might be missing?

vasartori · 18 July 2024 16:42

@troyanov I don’t think so. It’s a fresh installation, with a new database, new regiond, and rackd. At this point, the networks have not been configured.

troyanov · 18 July 2024 17:17

@vasartori thats interesting. I am trying try reproduce it now with jammy and ppa:maas/3.5. Will get back with results soon

vasartori · 18 July 2024 17:27

Also, I put the regiond in debug mode, but without luck. I’m not able to see which file or whatever is not found

If I access the regiond server directly, the error still appears (just to discard any misconfig with HAProxy)."

troyanov · 18 July 2024 17:30

Hm, it works for me.

Setup is just the default one:

lxc launch ubuntu:jammy maas-tester
apt-add-repository -y ppa:maas/3.5 && apt update -y && apt install -y maas

vasartori · 18 July 2024 17:50

My setup is a little bit more complex

This is my environment (sorry for the ugly diagram)

+----------------------------------------------------+                                                                                                                            
|    +--------------+            +--------------+    |                                                                                                                            
|    |              |            |              |    |                                                                                                                            
|    |   Database   -------------|   RegionD    |    |        +------------------------------------+                                                                              
|    |              |            |              |    |        |  +----------+                      |                                                                              
|    +--------------+            +-------|------+    |        |  |          |                      |                                                                              
|                                        |           |        |  |  Rackd   |  Physical Servers    |                                                                              
|                                        |           |        |  |          |                      |                                                                              
|                                +--------------+    |        |  +--|-------+                      |                                                                              
|                                |   HA Proxy   |    |        |     |                              |                                                                              
|                                |              --------------------+                              |                                                                              
|                                +---|----------+    |        |                           VLAN Y   |                                                                              
| VLAN X                             |               |        +------------------------------------+                                                                              
+------------------------------------|---------------+                                                                                                                            
                                     |                                                                                                                                            
                                     |                                                                                                                                            
                                     |                                                                                                                                            
                                     |                                                                                                                                            
                             Users  and/or Automations

Regiond, HAProxy, and Rackd each have two dedicated VMs, while the database has one.

troyanov · 18 July 2024 18:09

I see…
How did you install your MAAS?
I guess you were not using maas meta package?

Can you please share apt-cache policy maas maas-* from the RegionD?

vasartori · 18 July 2024 18:24

I’ll suppress the database setup

Regiond:

Add PPA Repo for maas 3.5
Install packages:
- chrony
- dnsutils
- tcpdump
Install MaaS Region package: maas-region-api
Create regiond.conf file
Migrate Database
Start maas-regiond service
Initialize the MaaS (create admin user)

To do these tasks, I’m using a ansible playbook (not the canonical playbook)

root@maas-regiond01:~#  apt-cache policy maas maas-*
maas:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-agent:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-cluster-controller:
  Installed: (none)
  Candidate: (none)
  Version table:
maas-region-controller:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-region-api:
  Installed: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
 *** 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
maas-rack-controller:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-dhcp:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-region-controller-min:
  Installed: (none)
  Candidate: (none)
  Version table:
maas-cli:
  Installed: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
 *** 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
maas-dns:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-netmon:
  Installed: (none)
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
     1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
maas-proxy:
  Installed: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
 *** 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
maas-common:
  Installed: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Candidate: 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1
  Version table:
 *** 1:3.5.0-16308-g.c799a1080-0ubuntu1~22.04.1 500
        500 https://ppa.launchpadcontent.net/maas/3.5/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
root@maas-regiond01:~#

troyanov · 18 July 2024 19:25

Hm, thats an interesting setup.
I was able to reproduce it and get the same error.

Need a bit more time to investigate further

troyanov · 18 July 2024 19:39

Okay, so the file that was not found is:
http+unix://%2Fvar%2Flib%2Fmaas%2Fapiserver-http.sock/api/v2/machines

And thats because maas-apiserver was not started.
Try service maas-apiserver start (and then service maas-regiond restart) and see if the issue is still there?

vasartori · 18 July 2024 20:13

@troyanov It Works!!!

Would be nice if this service was a dependency of the maas-regiond service. It makes sense?

troyanov · 18 July 2024 20:34

Sure thing.
I’ve filed a bug and the fix is coming.

system · 20 July 2024 20:35

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.