Automatic tag not working

Users
,
1

I am trying to set up automatic tagging on Maas 3.3. Below is my tag definition for “Gold”, this was done through the UI. After creating the tag, I was expecting all the machine matching the definition to be tagged automatically. I even recommission and redeploy a machine. The tag does not show up.

Anyone experience this?

Thanks

–
root@maas-2:~# maas admin tags read
Success.
Machine-readable output follows:
[
{
“name”: “virtual”,
“definition”: “”,
“comment”: “”,
“kernel_opts”: “”,
“resource_uri”: “/MAAS/api/2.0/tags/virtual/”
},
{
“name”: “Gold”,
“definition”: “//node[@class=“system”]/vendor = “Supermicro””,
“comment”: “This is a test comment.”,
“kernel_opts”: “”,
“resource_uri”: “/MAAS/api/2.0/tags/Gold/”
}
]

This is the Xpath installation log.

<lshw:node id="choice-squid" claimed="true" class="system" handle="DMI:0001">
 <lshw:description>Computer</lshw:description>
 <lshw:product>SSG-6019P-ACR12L+ (To be filled by O.E.M.)</lshw:product>
 <lshw:vendor>Supermicro</lshw:vendor>
 <lshw:version>0123456789</lshw:version>

–

2

Hi @johnliaotvc!

Nice to hear that you are playing with this feature! My gut feeling is that the definition might not be correct. IIUC your request, is //lshw:node[@class = 'system' and lshw:vendor[text() = 'Supermicro']] the expression you are looking for?

Note also that Once an automatic tag is created the screen will initially show that 0 machines are tagged. That is because MAAS is running a background task to auto-apply the tag to matching machines. It can take some time to see that the number of machines tagged is populating. as per documentation https://maas.io/docs/how-to-tag-machines#heading--how-to-create-automatic-tags

3

Or something like this one will also work:

//lshw:node[@class='system']/vendor[text()='Supermicro']

@johnliaotvc MAAS is using lxml so you can get a playground to validate your XPath selectors easily:

from lxml import etree
from io import StringIO
tree = etree.parse(StringIO('your XML goes here'))
tree.xpath('your XPath selector')
4

Thanks for the clarification. I created the following tags but none of the machines show up with the autotag. It has been a few hours. I am not sure how to get lxml to validate my XPath selector.

root@maas-2:~# maas admin tags read
Success.
Machine-readable output follows:
[
    {
        "name": "virtual",
        "definition": "",
        "comment": "",
        "kernel_opts": "",
        "resource_uri": "/MAAS/api/2.0/tags/virtual/"
    },
    {
        "name": "Gold",
        "definition": "//lshw:node[@class='system']/vendor[text()='Supermicro']",
        "comment": "This is a test comment.",
        "kernel_opts": "",
        "resource_uri": "/MAAS/api/2.0/tags/Gold/"
    },
    {
        "name": "Silver",
        "definition": "//lshw:node[@class='system']/lshw:vendor = 'Supermicro'",
        "comment": "",
        "kernel_opts": "",
        "resource_uri": "/MAAS/api/2.0/tags/Silver/"
    },
    {
        "name": "Bronze",
        "definition": "//lshw:node[@class='system']/lshw:vendor[text()='Supermicro']",
        "comment": "",
        "kernel_opts": "",
        "resource_uri": "/MAAS/api/2.0/tags/Bronze/"
    },
    {
        "name": "Plat",
        "definition": "//lshw:node[@class = 'system' and lshw:vendor[text() = 'Supermicro']]",
        "comment": "",
        "kernel_opts": "",
        "resource_uri": "/MAAS/api/2.0/tags/Plat/"
    }
]
5

@johnliaotvc

Sorry, I missed that you have lshw namespace as a prefix. IIRC this should not be there.

I just tried //node[@class='system']/vendor = 'HP' to capture machine with the following output and it worked for me:

<lshw:node id="pidgey" claimed="true" class="system" handle="DMI:0006">
 <lshw:description>Rack Mount Chassis</lshw:description>
 <lshw:product>ProLiant DL360 Gen9 (780018-S01)</lshw:product>
 <lshw:vendor>HP</lshw:vendor>
...
</lshw:node>
6

That was my initial autotag. This still doesn’t work for me.

//node[@class='system']/vendor = 'Supermicro'
7

@johnliaotvc I apologize for misreading your initial XPath selector.

Based on the description you provided everything should work. Can you please tell which MAAS version you are using and if it is a snap or deb package? I will try it locally on the same version as you have.

Sorry you’re having trouble there.

8

I am on snap version 3.3.3.

Thanks

9

Could you also check if you have any error message on the rack(s) ? Any relevant error message there might be useful for us to triage the problem.

Just to give you a bit of background, the matching is performed at rack(s) level and the result is sent back to the region with an HTTP call.

The logs are under

  1. /var/snap/maas/common/log/rackd.log if you are using snap
  2. /var/log/maas/rackd.log if you are using deb
10

I see this in the log regarding EvaluatingTag. Probably a bug? What version do you have that’s working? I tried out 3.4 beta 1 as well.

Thanks

2023-05-23 21:37:17 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'EvaluateTag' command. [maas-2:pid=268944:cmd=EvaluateTag:ask=6]
        Traceback (most recent call last):
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
            self.runUntilCurrent()
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback(  # type: ignore[misc]
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1138, in checkKnownErrors
            key = error.trap(*command.allErrors)
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
            result = inContext.theWork()  # type: ignore[attr-defined]
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
            inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/27405/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
            return func(*args, **kw)
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/rpc/tags.py", line 39, in evaluate_tag
            process_node_tags(
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/tags.py", line 339, in process_node_tags
            process_all(
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/tags.py", line 303, in process_all
            nodes_matched, nodes_unmatched = classify(
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/utils/__init__.py", line 123, in classify
            for ident, subject in subjects:
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/tags.py", line 279, in gen_node_details
            for system_id, details in get_details(batch).items():
          File "/snap/maas/27405/lib/python3.10/site-packages/provisioningserver/tags.py", line 78, in get_details_for_nodes
            data = process_response(client.get(path, op="details"))
          File "/snap/maas/27405/lib/python3.10/site-packages/apiclient/maas_client.py", line 270, in get
            return self.dispatcher.dispatch_query(
          File "/snap/maas/27405/lib/python3.10/site-packages/apiclient/maas_client.py", line 122, in dispatch_query
            res = opener.open(req)
          File "/usr/lib/python3.10/urllib/request.py", line 525, in open
          response = meth(req, response)
          File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
            response = self.parent.error(
          File "/usr/lib/python3.10/urllib/request.py", line 563, in error
            return self._call_chain(*args)
          File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
            result = func(*args)
          File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
            raise HTTPError(req.full_url, code, msg, hdrs, fp)
        urllib.error.HTTPError: HTTP Error 403: Forbidden
1 Like
11

I’ve just checked maas 3.3.3-13184-g.3e9972c19 from snap and everything works almost instantly.

What type of setup do you have? Rack and region are installed on the same machine?

12

Correct, rack and region server on the same machine.

13

Thats interesting.

Can you tell us a bit more about your setup? Maybe you are using an external reverse proxy?

As @r00ta mentioned, for the tag assignment rackd is using HTTP call towards regiond and for some reason it is getting HTTP 403.

src/provisioningserver/tags.py:105

path = f"/MAAS/api/2.0/tags/{tag_name}/"
    log.debug(
        "Updating nodes for {name}, adding {adding} removing {removing}",
        name=tag_name,
        adding=added,
        removing=removed,
    )
14

It is basic vanilla setup. Using the built-in MAAS proxy. I have included the output for both the region and rack controllers. I am using 8.8.8.8 as a DNS resolver. Is it using that to look up the regiond or rackd?

Screenshot 2023-05-24 at 5.30.14 PM
Screenshot 2023-05-24 at 5.30.14 PM767×323 14.9 KB

Here is the region-controllers output.

root@maas-2:~# maas admin region-controllers read
Success.
Machine-readable output follows:
[
    {
        "commissioning_status_name": "Passed",
        "distro_series": "jammy",
        "power_type": "",
        "system_id": "6df6gc",
        "network_test_status_name": "Unknown",
        "zone": {
            "name": "default",
            "description": "",
            "id": 1,
            "resource_uri": "/MAAS/api/2.0/zones/default/"
        },
        "power_state": "unknown",
        "other_test_status": -1,
        "osystem": "ubuntu",
        "fqdn": "maas-2.maas",
        "node_type_name": "Region and rack controller",
        "memory_test_status": -1,
        "ip_addresses": [
            "10.80.224.98"
        ],
        "current_installation_result_id": null,
        "version": "3.3.3-13184-g.3e9972c19",
        "node_type": 4,
        "architecture": "amd64/generic",
        "hostname": "maas-2",
        "interface_test_status_name": "Unknown",
        "status_action": "",
        "interface_test_status": -1,
        "swap_size": null,
        "hardware_info": {
            "system_vendor": "VMware, Inc.",
            "system_product": "VMware Virtual Platform",
            "system_family": "Unknown",
            "system_version": "Unknown",
            "system_sku": "Unknown",
            "system_serial": "VMware-42 33 ae 30 86 ae 24 b1-08 74 44 88 55 a9 72 22",
            "cpu_model": "Intel(R) Xeon(R) Gold 6130 CPU",
            "mainboard_vendor": "Intel Corporation",
            "mainboard_product": "440BX Desktop Reference Platform",
            "mainboard_serial": "Unknown",
            "mainboard_version": "Unknown",
            "mainboard_firmware_vendor": "Phoenix Technologies LTD",
            "mainboard_firmware_date": "12/12/2018",
            "mainboard_firmware_version": "6.00",
            "chassis_vendor": "No Enclosure",
            "chassis_type": "Other",
            "chassis_serial": "Unknown",
            "chassis_version": "N/A"
        },
        "cpu_test_status": -1,
        "description": "",
        "memory": 16384,
        "cpu_count": 8,
        "domain": {
            "authoritative": true,
            "ttl": null,
            "resource_record_count": 1,
            "name": "maas",
            "is_default": true,
            "id": 0,
            "resource_uri": "/MAAS/api/2.0/domains/0/"
        },
        "current_testing_result_id": null,
        "memory_test_status_name": "Unknown",
        "storage_test_status_name": "Unknown",
        "tag_names": [
            "virtual"
        ],
        "other_test_status_name": "Unknown",
        "testing_status": -1,
        "current_commissioning_result_id": 1,
        "commissioning_status": 2,
        "testing_status_name": "Unknown",
        "network_test_status": -1,
        "storage_test_status": -1,
        "interface_set": [
            {
                "links": [
                    {
                        "id": 1,
                        "mode": "static",
                        "ip_address": "10.80.224.98",
                        "subnet": {
                            "name": "10.80.224.0/21",
                            "description": "",
                            "vlan": {
                                "vid": 0,
                                "mtu": 1500,
                                "dhcp_on": true,
                                "external_dhcp": null,
                                "relay_vlan": null,
                                "primary_rack": "6df6gc",
                                "secondary_rack": null,
                                "space": "undefined",
                                "name": "untagged",
                                "fabric": "fabric-0",
                                "fabric_id": 0,
                                "id": 5001,
                                "resource_uri": "/MAAS/api/2.0/vlans/5001/"
                            },
                            "cidr": "10.80.224.0/21",
                            "rdns_mode": 2,
                            "gateway_ip": "10.80.224.1",
                            "dns_servers": [],
                            "allow_dns": true,
                            "allow_proxy": true,
                            "active_discovery": false,
                            "managed": true,
                            "disabled_boot_architectures": [],
                            "space": "undefined",
                            "id": 1,
                            "resource_uri": "/MAAS/api/2.0/subnets/1/"
                        }
                    }
                ],
                "type": "physical",
                "vlan": {
                    "vid": 0,
                    "mtu": 1500,
                    "dhcp_on": true,
                    "external_dhcp": null,
                    "relay_vlan": null,
                    "primary_rack": "6df6gc",
                    "secondary_rack": null,
                    "space": "undefined",
                    "name": "untagged",
                    "fabric": "fabric-0",
                    "fabric_id": 0,
                    "id": 5001,
                    "resource_uri": "/MAAS/api/2.0/vlans/5001/"
                },
                "numa_node": 0,
                "system_id": "6df6gc",
                "name": "ens160",
                "enabled": true,
                "tags": [],
                "firmware_version": null,
                "id": 1,
                "children": [],
                "sriov_max_vf": 0,
                "mac_address": "00:50:56:b3:e2:fd",
                "effective_mtu": 1500,
                "parents": [],
                "product": null,
                "params": "",
                "vendor": null,
                "link_connected": true,
                "interface_speed": 10000,
                "discovered": null,
                "link_speed": 10000,
                "resource_uri": "/MAAS/api/2.0/nodes/6df6gc/interfaces/1/"
            }
        ],
        "cpu_speed": 2100,
        "cpu_test_status_name": "Unknown",
        "hardware_uuid": "30ae3342-ae86-b124-0874-448855a97222",
        "resource_uri": "/MAAS/api/2.0/regioncontrollers/6df6gc/"
    }
]

Here is the rack controller output.

root@maas-2:~# maas admin rack-controllers read
Success.
Machine-readable output follows:
[
    {
        "commissioning_status_name": "Passed",
        "distro_series": "jammy",
        "power_type": "",
        "system_id": "6df6gc",
        "network_test_status_name": "Unknown",
        "zone": {
            "name": "default",
            "description": "",
            "id": 1,
            "resource_uri": "/MAAS/api/2.0/zones/default/"
        },
        "power_state": "unknown",
        "other_test_status": -1,
        "osystem": "ubuntu",
        "fqdn": "maas-2.maas",
        "node_type_name": "Region and rack controller",
        "memory_test_status": -1,
        "service_set": [
            {
                "name": "ntp_region",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "proxy",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "bind9",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "reverse_proxy",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "regiond",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "syslog_region",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "dns_rack",
                "status": "unknown",
                "status_info": "managed by the region"
            },
            {
                "name": "dhcpd6",
                "status": "off",
                "status_info": ""
            },
            {
                "name": "proxy_rack",
                "status": "unknown",
                "status_info": "managed by the region"
            },
            {
                "name": "dhcpd",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "tftp",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "syslog_rack",
                "status": "unknown",
                "status_info": "managed by the region"
            },
            {
                "name": "ntp_rack",
                "status": "unknown",
                "status_info": "managed by the region"
            },
            {
                "name": "http",
                "status": "running",
                "status_info": ""
            },
            {
                "name": "rackd",
                "status": "running",
                "status_info": ""
            }
        ],
        "ip_addresses": [
            "10.80.224.98"
        ],
        "current_installation_result_id": null,
        "version": "3.3.3-13184-g.3e9972c19",
        "node_type": 4,
        "architecture": "amd64/generic",
        "hostname": "maas-2",
        "interface_test_status_name": "Unknown",
        "status_action": "",
        "interface_test_status": -1,
        "swap_size": null,
        "hardware_info": {
            "system_vendor": "VMware, Inc.",
            "system_product": "VMware Virtual Platform",
            "system_family": "Unknown",
            "system_version": "Unknown",
            "system_sku": "Unknown",
            "system_serial": "VMware-42 33 ae 30 86 ae 24 b1-08 74 44 88 55 a9 72 22",
            "cpu_model": "Intel(R) Xeon(R) Gold 6130 CPU",
            "mainboard_vendor": "Intel Corporation",
            "mainboard_product": "440BX Desktop Reference Platform",
            "mainboard_serial": "Unknown",
            "mainboard_version": "Unknown",
            "mainboard_firmware_vendor": "Phoenix Technologies LTD",
            "mainboard_firmware_date": "12/12/2018",
            "mainboard_firmware_version": "6.00",
            "chassis_vendor": "No Enclosure",
            "chassis_type": "Other",
            "chassis_serial": "Unknown",
            "chassis_version": "N/A"
        },
        "cpu_test_status": -1,
        "description": "",
        "memory": 16384,
        "cpu_count": 8,
        "domain": {
            "authoritative": true,
            "ttl": null,
            "resource_record_count": 1,
            "name": "maas",
            "is_default": true,
            "id": 0,
            "resource_uri": "/MAAS/api/2.0/domains/0/"
        },
        "current_testing_result_id": null,
        "memory_test_status_name": "Unknown",
        "storage_test_status_name": "Unknown",
        "tag_names": [
            "virtual"
        ],
        "other_test_status_name": "Unknown",
        "testing_status": -1,
        "current_commissioning_result_id": 1,
        "commissioning_status": 2,
        "testing_status_name": "Unknown",
        "network_test_status": -1,
        "storage_test_status": -1,
        "interface_set": [
            {
                "links": [
                    {
                        "id": 1,
                        "mode": "static",
                        "ip_address": "10.80.224.98",
                        "subnet": {
                            "name": "10.80.224.0/21",
                            "description": "",
                            "vlan": {
                                "vid": 0,
                                "mtu": 1500,
                                "dhcp_on": true,
                                "external_dhcp": null,
                                "relay_vlan": null,
                                "primary_rack": "6df6gc",
                                "secondary_rack": null,
                                "space": "undefined",
                                "name": "untagged",
                                "fabric": "fabric-0",
                                "fabric_id": 0,
                                "id": 5001,
                                "resource_uri": "/MAAS/api/2.0/vlans/5001/"
                            },
                            "cidr": "10.80.224.0/21",
                            "rdns_mode": 2,
                            "gateway_ip": "10.80.224.1",
                            "dns_servers": [],
                            "allow_dns": true,
                            "allow_proxy": true,
                            "active_discovery": false,
                            "managed": true,
                            "disabled_boot_architectures": [],
                            "space": "undefined",
                            "id": 1,
                            "resource_uri": "/MAAS/api/2.0/subnets/1/"
                        }
                    }
                ],
                "type": "physical",
                "vlan": {
                    "vid": 0,
                    "mtu": 1500,
                    "dhcp_on": true,
                    "external_dhcp": null,
                    "relay_vlan": null,
                    "primary_rack": "6df6gc",
                    "secondary_rack": null,
                    "space": "undefined",
                    "name": "untagged",
                    "fabric": "fabric-0",
                    "fabric_id": 0,
                    "id": 5001,
                    "resource_uri": "/MAAS/api/2.0/vlans/5001/"
                },
                "numa_node": 0,
                "system_id": "6df6gc",
                "name": "ens160",
                "enabled": true,
                "tags": [],
                "firmware_version": null,
                "id": 1,
                "children": [],
                "sriov_max_vf": 0,
                "mac_address": "00:50:56:b3:e2:fd",
                "effective_mtu": 1500,
                "parents": [],
                "product": null,
                "params": "",
                "vendor": null,
                "link_connected": true,
                "interface_speed": 10000,
                "discovered": null,
                "link_speed": 10000,
                "resource_uri": "/MAAS/api/2.0/nodes/6df6gc/interfaces/1/"
            }
        ],
        "cpu_speed": 2100,
        "cpu_test_status_name": "Unknown",
        "hardware_uuid": "30ae3342-ae86-b124-0874-448855a97222",
        "resource_uri": "/MAAS/api/2.0/rackcontrollers/6df6gc/"
    }
]
15

Everything indeed looks like a common setup.

I am going to try couple more things, but meanwhile can you please check other logs for errors:
/var/snap/maas/common/log/http/error.log
(and also check access.log just to ensure request is there)

And maybe there is something interesting in /var/snap/maas/common/log/regiond.log

16

@johnliaotvc can you please share the contents of /var/snap/maas/current/rackd.conf?

17

There’s nothing interesting in the logs. Here is the rackd.conf. A one liner.

root@maas-2:/var/snap/maas/current# more rackd.conf
maas_url: http://10.80.224.98:5240/MAAS

18

Hello @johnliaotvc

So far I am not able to reproduce it and I don’t have good ideas what could be wrong here.
The rest of communication between rack and region works fine? Thats the only feature that doesn’t work?

19

I added 22 identical machine into MAAS and two of them pick up the automatic tags. Not sure how to debug this. The installation output looks the same on both.

20

I get an almost identical stack trace with HTTP 403 when evaluating tags on MaaS 3.3.4-13189-g.f88272d1e. updated from 2.8 via packages.

To rule out issues with my XPath expression I tested with the expression true() as seen in the unit tests, too.

This is a single node installation with all components running on Ubuntu 22.04.

In the http/access.log I see two successful requests /MAAS/api/2.0/nodes/(id of device)/?op=details

followed by one unsuccessful aka. 403 request to /MAAS/api/2.0/nodes/(id of a machine)/?op=details

edit: the “true()” tag has since been applied to the controller

and while playing around I sometimes get a different order. 403 appears to happen when trying to access a machine that has an owner (allocated or deployed), a machine without owner got a 200, but was directly followed by a machine with an owner again.