MAAS crashing after a couple of days

Users
, ,
1

My snap installed MAAS web UI is showing this error after a few days of being left alone

builtins.OSError: [Errno 24] Too many open files: ‘/snap/maas/18199/usr/share/maas/web/static/index.html’

The problem can be cleared by running
sudo snap restart maas.supervisor

What is a good way to diagnose this? Thanks.

2

So its happening more often now, a few hours after a restart.

This is the snap version 3.1.0.

When this happens I see:

  • Failed to contact region. in the rackd log.
  • The regiond log seems to show no errors.
  • The maas logs show errors connecting to the power webhook (because my webhook server is offline right now).
  • named.log is showing many errors I don’t understand.

Logs attached.

https://drive.google.com/file/d/1by0Ds7MsqgT2G2-Pu_BO0_KJssdlo98-/view?usp=sharing

3

Hi @gilesknap,

you seem to have a DNS configuration issue that is exhausting the TCP ports inside the snap jail. You can try going to Settings > Network > DNS and disabling DNSSEC, or trying a different upstream server.

4

Awesome thanks. I was using Open DNS and have switched to Google since I don’t really need the filtering for my MAAS devices. Will report on the results.

5

Crashed again when I looked this evening (with DNS upstream as 8.8.8.8 8.8.4.4).
https://drive.google.com/file/d/12u8cxs2SHeJtaw64L3VKm850MkGIkKFN/view?usp=sharing

Will now try disable DNSSEC.

6

I’ve switched to Google for upstream DNS and turned off DNSSEC. But still have MAAS crashing after several hours running.

I’ve noticed that the server on which I’m running seems to be pointing at itself for DNS and that seems a bit wrong (192.168.0.10 is the server’s own IP) :


giles@vmaas1:~$ systemd-resolve --status
Global
       LLMNR setting: no                  
MulticastDNS setting: no                  
  DNSOverTLS setting: no                  
      DNSSEC setting: no                  
    DNSSEC supported: no                  
  Current DNS Server: 192.168.0.10        
         DNS Servers: 192.168.0.10        
Fallback DNS Servers: 8.8.8.8             
          DNS Domain: maas                
          DNSSEC NTA: 10.in-addr.arpa     
                      16.172.in-addr.arpa 
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa 
                      18.172.in-addr.arpa 
...

I’ve done apply netplan with the following yaml but this has not changed the above

giles@vmaas1:~$ cat /etc/netplan/00-installer-config.yaml 
# This is the network config written by 'subiquity'
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      dhcp4: false
      addresses: [192.168.0.10/24]
      gateway4: 192.168.0.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

Is this a problem or just a red herring?

7

I don’t think it is wrong to point the host at itself for resolution if you are running a DNS resolver such as MAAS. In fact, this is the easiest way for you to be able to resolve maas managed domain names from the maas server.

see this diagram from this thread:

image
image961×613 22.9 KB

I know I’m not answering your quesion/issue, just commenting on the dns setup.

8

Thanks Anton, now I need another theory as to what the problem is :slight_smile:

9

This looks to be resolved.

I had a rogue copy of K3S Agent running in the VM alongside MAAS. I must have accidentally installed it. I’m not sure of the exact problem but removing that seems to have fixed it.

1 Like
10

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.