Conflicting virsh resources, gui vs cli

Users
,
1

Issue:
Virsh over-commit are not consistent, main issues are as follows:

  • Resources are not detected in general
  • GUI and CLI not reporting correctly - They maintain at 1.0 over commit.

Resources:

  • MAAS: 3.2.0
  • Dell R630 with:
    • 40 cores
    • 128 GB RAM
    • 1 TB Storage
    • Ubuntu 20.04, deployed as KVM-virsh host.

Resources not being detected:
Upon first deploying the server with virsh KVM, no cores/RAM were detected in the CLI, but the GUI reported as follows:
5.0 x CPU over commit & 5.0x Memory over commit.
0 of 200 allocated (cores)
0 of 640GiB allocated (RAM)
0 of 983.4GB allocated (DISK)

On CLI, my resources for the same host are:
“cpu_over_commit_ratio”: 5.0,
“memory_over_commit_ratio”: 5.0,
“total”: {
“cores”: 0,
“memory”: 0,
“local_storage”: 983428714496
}

At this point, I was concerned maybe there was a BIOS setting that wasn’t right disabling the virtualization, so I rebooted, verified (no changes made). After reboot, I changed my resource over commit and have this issue:

GUI and CLI not reporting correctly:
I then modified my CPU overcommit to 0.1 & kept my memory over commit to 5.0. At this point the CPU cores were finally detected.
GUI:
0 of 4 allocated cores
0 of 640GiB allocated memory
0 of 983.4GB allocated storage

“cpu_over_commit_ratio”: 0.1,
“memory_over_commit_ratio”: 5.0,
“total”: {
“cores”: 40,
“memory”: 128818,
“local_storage”: 983428714496
},

Changed cores to 2.0
“cpu_over_commit_ratio”: 2.0,
“memory_over_commit_ratio”: 5.0,
“total”: {
“cores”: 40,
“memory”: 128818,
“local_storage”: 983428714496
},

Notes:
It appears that the GUI reports what the numbers should be correctly, however, the CLI either reports 0 cores and 0 ram OR only the number of cores and ram the system has. At times, it appears that the CLI will report 0 total resources as well. Also, if a VM is deployed, it can’t seem to commission due to “Unable to connect to any rack controller r8ydny; no connections available.” The networks on the host have bridges to all of the networks with the VM being on all of them. I’d love to dig into this problem further if anyone is willing as I don’t think this was an issue on 2.8 (the version we used prior to this).

2

Hi James,

do the cores get detected if you do a refresh of the VM host?

3

Hi Bjornt,

No, refreshing does not change anything.

4 
$ maas xxxxx vm-host update 39 cpu_over_commit_ratio=10 memory_over_commit_ratio=10
    Success.
    Machine-readable output follows:
    {
        "name": "R1-710-26",
        "architectures": [
            "amd64/generic"
        ],
        "zone": {
            "name": "default",
            "description": "",
            "id": 1,
            "resource_uri": "/MAAS/api/2.0/zones/default/"
        },
        "capabilities": [
            "composable",
            "dynamic_local_storage",
            "over_commit",
            "storage_pools"
        ],
        "total": {
            "cores": 16,
            "memory": 145035,
            "local_storage": 1967300759552
        },
        "id": 39,
        "storage_pools": [
            {
                "id": "baa103c3-d7e6-4037-9469-28be8e9fd269",
                "name": "maas",
                "type": "dir",
                "path": "/var/lib/libvirt/maas-images",
                "total": 1967300759552,
                "used": 0,
                "available": 1967300759552,
                "default": true
            }
        ],
        "tags": [
            "pod-console-logging",
            "virtual"
        ],
        "available": {
            "cores": 16,
            "memory": 145035,
            "local_storage": 1967300759552
        },
        "memory_over_commit_ratio": 10.0,
        "used": {
            "cores": 0,
            "memory": 0,
            "local_storage": 0
        },
        "host": {
            "system_id": "hxq3k4",
            "__incomplete__": true
        },
        "default_macvlan_mode": "",
        "version": "6.0.0",
        "pool": {
            "name": "virtual_machine_pool",
            "description": "",
            "id": 1,
            "resource_uri": "/MAAS/api/2.0/resourcepool/1/"
        },
        "type": "virsh",
        "cpu_over_commit_ratio": 10.0,
        "resource_uri": "/MAAS/api/2.0/vm-hosts/39/"
    }
5 
$ maas xxxxx vm-host refresh 39
Success.
Machine-readable output follows:
{
    "name": "R1-710-26",
    "architectures": [
        "amd64/generic"
    ],
    "zone": {
        "name": "default",
        "description": "",
        "id": 1,
        "resource_uri": "/MAAS/api/2.0/zones/default/"
    },
    "capabilities": [
        "composable",
        "dynamic_local_storage",
        "over_commit",
        "storage_pools"
    ],
    "total": {
        "cores": 16,
        "memory": 145035,
        "local_storage": 1967300759552
    },
    "id": 39,
    "storage_pools": [
        {
            "id": "baa103c3-d7e6-4037-9469-28be8e9fd269",
            "name": "maas",
            "type": "dir",
            "path": "/var/lib/libvirt/maas-images",
            "total": 1967300759552,
            "used": 0,
            "available": 1967300759552,
            "default": true
        }
    ],
    "tags": [
        "pod-console-logging",
        "virtual"
    ],
    "available": {
        "cores": 16,
        "memory": 145035,
        "local_storage": 1967300759552
    },
    "memory_over_commit_ratio": 10.0,
    "used": {
        "cores": 0,
        "memory": 0,
        "local_storage": 0
    },
    "host": {
        "system_id": "hxq3k4",
        "__incomplete__": true
    },
    "default_macvlan_mode": "",
    "version": "6.0.0",
    "pool": {
        "name": "virtual_machine_pool",
        "description": "",
        "id": 1,
        "resource_uri": "/MAAS/api/2.0/resourcepool/1/"
    },
    "type": "virsh",
    "cpu_over_commit_ratio": 10.0,
    "resource_uri": "/MAAS/api/2.0/vm-hosts/39/"
}
6

@james-o-benson, i can’t reproduce this problem using LXD VMs and a LXD VM host, or my Minis Forum NUC network (using a 4 core NUC as a KVM host). can you try this with different hardware and see if you can reproduce it, or try it with an LXD KVM?

7

@james-o-benson, this item is marked as “bug-filed” – did you file a bug? if so, what’s the bug number?

8

@billwear, LXD seems to function properly. It is the virsh issue. Also, when I create a virsh VM, the commissioning fails.

LXD hosts seem to work perfectly fine.

9

@billwear. My mistake a bug was not filed. I have no tracking number.

10

okay, do you have to use virsh, or can you just switch to LXD? bug should still be filed, but maybe your problem will be solved itmt.

11

For what we are doing, virsh is preferred. We are doing automated Openstack test deployments and need nested virtualization which I can’t seem to get working properly with LXD. In MaaS 2.8 (maybe 2.9), prior to our upgrade to 3.2, virsh was working perfectly for our use-case.

1 Like
12

okay, and you didn’t change anything in your virsh setup when you moved to 3.2? if not, you should file a bug. something has gone wonky, and using this guide to file a bug will help us find it, even if it turns out to be some other kind of issue.

13

Correct, nothing has changed in our setup/deployment. I’ll try to open a bug this afternoon.

1 Like
14

thanks. this one’s weird at the moment, but virsh is sometimes…difficult, and not because of the setup or config.

15
1 Like
16

@billwear, I found that while the HWE kernel does not support nested virtualization in LXD, the edge kernel does support it. So the bug still exists, but at least I have a work around for the time being.

1 Like