MAAS 3.2.9 creates Calico Interfaces 80.000 fabrics

I could no longer customise the network configuration of individual servers or display the list of fabrics. After some searching I was able to find out that fabrics were created in the database every 15 minutes. As a result, the database table fabrics contains about 80t data records and the maas interface crashes on loading. BTW the event table has 14.000.000 entrys

Maas manages four VMs running a K8 cluster, with a master and worker node running on two machines each. Apparently the hardware sync recognises Calico interfaces from the K8 cluster every 15 minutes and creates a fabric for each interface.

Is it possible to avoid this behaviour?

Name Version Rev Tracking Publisher Notes
maas 3.2.9-12055-g.c3d5597a7 31022 3.2/stable canonical✓ -

Hi @erikdy ,

this sounds like a bug to me. Could you please open it according to this guide.

Thank you very much, appreciate it!

Sounds like you have enabled the periodic hardware sync option. I too made this mistake a while ago.

If you disable it and then try and clean up the fabircs you should be o.k

Hi,
That’s what I think too.
But how can I deactivate this? In the UI I only found a field that allows me to set the interval in minutes. Setting it to 0 doesn’t work and anything else like 9999999 min would be a weird solution.

So the only way to disabled it i think is to re-enlist the machine with it disabled.
but, what you can do is disable and remove the system service that is running on the nodes.

it will then not report info. The problem your going to have is removing the current networks and fabrics.

You might have to give up with this MAAS install, and look to start again.

that would need some thought and research, but it is possible to import built servers into maas.

Hi,

I found a fix for my system.

WARNING: POTENTIAL SYSTEM DAMAGE
– Use this solution at your own risk. Ensure a backup and understand the commands before executing.

I used pgAdmin to disable the hardware sync flag in the maasserver_node table.
To clear the fabrics and vlan table:

DELETE FROM maasserver_vlan
WHERE maasserver_vlan.id NOT IN (
    SELECT maasserver_vlan.id 
    FROM maasserver_vlan
    LEFT JOIN maasserver_interface ON maasserver_vlan.id = maasserver_interface.vlan_id
    WHERE maasserver_interface.vlan_id IS NOT NULL i 
)
AND maasserver_vlan.id NOT IN (
    SELECT maasserver_vlan.id 
    FROM maasserver_vlan
    JOIN maasserver_subnet ON maasserver_vlan.id = maasserver_subnet.vlan_id
);

DELETE FROM maasserver_fabric
WHERE maasserver_fabric.id NOT IN (
    SELECT maasserver_fabric.id 
    FROM maasserver_fabric
    LEFT JOIN maasserver_vlan ON maasserver_vlan.fabric_id = maasserver_fabric.id
    WHERE maasserver_vlan.fabric_id IS NOT NULL
);

It works for now, but I hope there will be a fix for the problem in the next release.

I’ve replicated the issue on 3.3.4. Here’s the full solution to disable periodic hardware sync by editing the database entries for the hosts on MaaS. (use at your own risk)

First, ssh to your MaaS controller. Then connect to the maasdb as the postgres user.

sudo su postgres
psql
postgres=# \c maasdb;

Then select the hosts by id and enable_hw_sync from the maasserver_node table to see what the current status is.

maasdb=# select id,enable_hw_sync from maasserver_node;
 id | enable_hw_sync 
----+----------------
  1 | f
  2 | f
  7 | f
  5 | t
  6 | t
  8 | t
(6 rows)

In my case, I want to disable hardware sync (set enable_hw_sync to false) for all of my hosts, so I used an UPDATE statement for all records in maasserver_node.

UPDATE maasserver_node SET enable_hw_sync = false;

There is also a system timer named maas_hardware_sync on each host to clean up. (I found that out by watching the fabrics get added in real-time after I thought I had cleaned them up.)

$ sudo systemctl list-units --type=timer maas_hardware_sync.timer
  UNIT                     LOAD   ACTIVE SUB     DESCRIPTION                                      
  maas_hardware_sync.timer loaded active waiting Timer for periodically running MAAS hardware sync

On each host, disable and remove the timer.

sudo systemctl disable maas_hardware_sync.timer
sudo systemctl stop maas_hardware_sync.timer
sudo rm /lib/systemd/system/maas_hardware_sync.timer

In my case, they were non-production hosts used for testing the infrastructure deployment; I chose to redeploy the nodes. Same affect.

Then, clean up the fabrics and VLANs. Using these SQL commands, two DELETE statements intended to remove records from the maasserver_vlan and maasserver_fabric tables.

maasdb=# DELETE FROM maasserver_vlan
	WHERE maasserver_vlan.id NOT IN (
	    SELECT maasserver_vlan.id 
	    FROM maasserver_vlan
	    LEFT JOIN maasserver_interface ON maasserver_vlan.id = maasserver_interface.vlan_id
	    WHERE maasserver_interface.vlan_id IS NOT NULL  
	)
	AND maasserver_vlan.id NOT IN (
	    SELECT maasserver_vlan.id 
	    FROM maasserver_vlan
	    JOIN maasserver_subnet ON maasserver_vlan.id = maasserver_subnet.vlan_id
	);

	DELETE FROM maasserver_fabric
	WHERE maasserver_fabric.id NOT IN (
	    SELECT maasserver_fabric.id 
	    FROM maasserver_fabric
	    LEFT JOIN maasserver_vlan ON maasserver_vlan.fabric_id = maasserver_fabric.id
	    WHERE maasserver_vlan.fabric_id IS NOT NULL
	);

Let’s break down each DELETE statement:

  1. DELETE FROM maasserver_vlan: This statement deletes VLANs (from the table maasserver_vlan) that are not associated with a network interface of a server (left join with maasserver_interface on the vlan_id) or used by a subnet (inner join with maasserver_subnet on the vlan_id).

  2. DELETE FROM maasserver_fabric: This statement deletes fabrics (from the table maasserver_fabric) that are not associated with a VLAN (left join with maasserver_vlan on the fabric_id).

In my case, that was 23902 VLANs and 23899 fabrics.

DELETE 23902
DELETE 23899

I had a few fabrics left over that I had to delete the subnet for first, but once I did and ran the DELETE statements again, my list was clear of extra fabrics.

1 Like

For reference the bug has been reported as Bug #2043970 " MAAS 3.2.9 creates for Calico Interfaces 80.000 f... : Bugs : MAAS

2 Likes