Trying to create VM in an LXD host. Even if there are a lot of memory and cores, it will not accept it.
Any ideas what could be wrong?
1 Like
Hi @bjornrun,
- what version of MAAS and LXD are you using?
- is the number of available cores and memory correct for this VM host?
- could you please look at logs (
/var/snap/maas/common/log/{maas,regiond,rackd}.log) for errors
I was on latest/edge and realised it is better to be on 3.1/stable for support. I have already switched to it. I’ll check if I get same problem on same LXD server…
The UI is different on 3.1/stable but the error is the same.
Log excerpt:
2022-02-02 17:41:45 metadataserver.api: [critical] ml36-lxd.maas(68rhbx): commissioning script ‘50-maas-01-commissioning’ failed during post-processing.
Traceback (most recent call last):
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/api.py”, line 817, in signal
target_status = process(node, request, status)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/api.py”, line 641, in _process_commissioning
self._store_results(
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/api.py”, line 529, in _store_results
script_result.store_result(
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/models/scriptresult.py”, line 376, in store_result
signal_status = try_or_log_event(
— —
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/api.py”, line 447, in try_or_log_event
func(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/hooks.py”, line 956, in process_lxd_results
_process_lxd_resources(node, data)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/hooks.py”, line 589, in _process_lxd_resources
network_devices = update_node_network_information(node, data, numa_nodes)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/hooks.py”, line 221, in update_node_network_information
update_node_interfaces(node, data)
File “/snap/maas/18199/lib/python3.8/site-packages/provisioningserver/utils/twisted.py”, line 202, in wrapper
result = func(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/utils/orm.py”, line 740, in call_within_transaction
return func_within_txn(*args, **kwargs)
File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
return func(*args, **kwds)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/network.py”, line 101, in update_node_interfaces
interface = update_interface(
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/network.py”, line 168, in update_interface
return update_child_interface(node, name, network, links)
File “/snap/maas/18199/lib/python3.8/site-packages/metadataserver/builtin_scripts/network.py”, line 391, in update_child_interface
interface = child_type.objects.get_or_create_on_node(
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/models/interface.py”, line 500, in get_or_create_on_node
interface = self.create(
File “/snap/maas/18199/usr/lib/python3/dist-packages/django/db/models/manager.py”, line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/snap/maas/18199/usr/lib/python3/dist-packages/django/db/models/query.py”, line 422, in create
obj.save(force_insert=True, using=self.db)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/models/interface.py”, line 1875, in save
super().save(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/models/interface.py”, line 1645, in save
return super().save(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/models/cleansave.py”, line 183, in save
self.full_clean(
File “/snap/maas/18199/usr/lib/python3/dist-packages/django/db/models/base.py”, line 1203, in full_clean
raise ValidationError(errors)
django.core.exceptions.ValidationError: {‘mac_address’: [‘This field cannot be blank.’]}2022-02-02 17:41:45 regiond: [info] 10.8.8.184 POST /MAAS/metadata/latest/ HTTP/1.1 --> 200 OK (referrer: -; agent: Python-urllib/3.8)
2022-02-02 17:41:49 regiond: [info] 10.8.8.183 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:42:04 regiond: [info] 10.8.8.184 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:42:19 regiond: [info] 10.8.8.183 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:42:34 regiond: [info] 10.8.8.184 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:42:49 regiond: [info] 10.8.8.183 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:42:51 regiond: [info] 100.82.220.113 GET /MAAS/accounts/login/ HTTP/1.1 --> 200 OK (referrer: http://10.8.8.183:5240/MAAS/r/settings; agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.175 Safari/537.36)
2022-02-02 17:43:04 regiond: [info] 10.8.8.184 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-02-02 17:43:09 maasserver.websockets.protocol: [critical] Error on request (108) pod.compose: {“cores”: [“Ensure this value is less than or equal to 0.”], “memory”: [“Ensure this value is less than or equal to 0.”]}
Traceback (most recent call last):
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 501, in errback
self._startRunCallbacks(fail)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 568, in _startRunCallbacks
self._runCallbacks()
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1475, in gotResult
_inlineCallbacks(r, g, status)
— —
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/internet/defer.py”, line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/python/failure.py”, line 491, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/websockets/handlers/pod.py”, line 317, in compose
form = await deferToDatabase(get_form, pod, params)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 250, in inContext
result = inContext.theWork()
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/python/threadpool.py”, line 266, in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/python/context.py”, line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File “/snap/maas/18199/usr/lib/python3/dist-packages/twisted/python/context.py”, line 85, in callWithContext
return func(*args,**kw)
File “/snap/maas/18199/lib/python3.8/site-packages/provisioningserver/utils/twisted.py”, line 870, in callInContext
return func(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/provisioningserver/utils/twisted.py”, line 202, in wrapper
result = func(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/utils/orm.py”, line 756, in call_within_transaction
return func_outside_txn(*args, **kwargs)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/utils/orm.py”, line 559, in retrier
return func(*args, **kwargs)
File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
return func(*args, **kwds)
File “/snap/maas/18199/lib/python3.8/site-packages/maasserver/websockets/handlers/pod.py”, line 307, in get_form
raise HandlerValidationError(form.errors)
maasserver.websockets.base.HandlerValidationError: {‘cores’: [‘Ensure this value is less than or equal to 0.’], ‘memory’: [‘Ensure this value is less than or equal to 0.’]}
Not sure if it is the same problem I had, but when I saw this I had to select the “Take action” button on the problematic KVM (in the UI) and select “Refresh”. Not sure what is causing it, but it looks like the metadata gets desync’d and prevents me from being able to compose VMs / machines as I need them.
HTH
- Bill
It seems related to how the LXD server is set up. If MAAS creates the LXD server, it just works. About to investigate the differences.
Same issue on 3.2.6… was able to work around it by bumping up the core/memory overcommit multipliers
1 Like
@dillon-n, can you run a pods read command and let me know the output? I’m specifically intertested in the memory_over_commit_ratio, used, available, and cpu_over_commit_ratio sections. Thank you!
@james-o-benson it doesn’t look like I have the pods command, is that an LXD-specific thing? I’m currently using libvirt due to a different bug MAAS KVM Host Fails Deployment w/error in rackd.log
No it should work with LXD or libvirt. I also can’t deploy LXD due to the timeout. And libvirt hosts deploys but commissioning the VM times out. I had the same error as above, what I noticed is that the GUI showed one thing but CLI showed another. See:
Ah I see, I was thinking pods was an actual executable command on the VM host, but it’s a subcommand on the MAAS CLI, which I haven’t delved into much yet. Here’s the relevant output you’re looking for:
{
"memory_over_commit_ratio": 1.3,
"cpu_over_commit_ratio": 4.0,
"version": "6.0.0",
"available": {
"cores": -1,
"memory": -2048,
"local_storage": 236466909184
},
"total": {
"cores": 0,
"memory": 0,
"local_storage": 244466909184
},
"used": {
"cores": 1,
"memory": 2048,
"local_storage": 8000000000
},
"type": "virsh",
}
The ratios there are the same as the ones I set in the UI. However available and total do NOT match, obviously the system has more than 0 cores and memory 
.
Really those numbers should be 20 and 160GiB respectively, which is how they display in the UI. I do have one VM deployed with 1C/2GiB, so used is correct. And it looks like available is just substracting used from total.
Storage is correct in all cases.
E: and I just realized I can’t create any more VMs than the one that’s already deployed, presumably because available is negative 
Try to do a ‘refresh’, you can do that in the GUI under that specific nodes actions in the KVM menu. For me, that fixed some of the resource issues, but also try to do a deploy. Because for me, I was able to create VMs, but none of them ever pass commissioning stage.
EDIT: That fixes the resources some of the time, but not always.
I am having the same issue on MAAS 3.2.6 using a refreshly deployed machine. Due to a bug in 3.2.6, I can’t use maas to initialize LXD on the host so I have to manually do it with LXD init.
VM Host:
$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, lvm, zfs, btrfs, ceph, cephobject) [default=zfs]: dir
Would you like to connect to a MAAS server? (yes/no) [default=no]: yes
What's the name of this host in MAAS? [default=smoothwall-lxc]:
URL of your MAAS server (e.g. http://1.2.3.4:5240/MAAS): http://10.160.1.10:5240/MAAS
API key for your MAAS server: <redacted>
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]: yes
Address to bind LXD to (not including port) [default=all]:
Port to bind LXD to [default=8443]:
Trust password for new clients:
Again:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
MAAS:
$ maas ben vm-hosts read | jq '.[] | select (.name=="smoothwall-lxc")'
{
"available": {
"cores": 0,
"memory": 0,
"local_storage": 471146233856
},
"capabilities": [
"composable",
"dynamic_local_storage",
"over_commit",
"storage_pools"
],
"total": {
"cores": 0,
"memory": 0,
"local_storage": 471146233856
},
"zone": {
"name": "default",
"description": "",
"id": 1,
"resource_uri": "/MAAS/api/2.0/zones/default/"
},
"architectures": [
"amd64/generic"
],
"cpu_over_commit_ratio": 1,
"memory_over_commit_ratio": 1,
"storage_pools": [
{
"id": "default",
"name": "default",
"type": "dir",
"path": "/var/snap/lxd/common/lxd/storage-pools/default",
"total": 471146233856,
"used": 0,
"available": 471146233856,
"default": true
}
],
"host": {
"system_id": "qnmbfd",
"__incomplete__": true
},
"tags": [
"pod-console-logging"
],
"version": "5.6",
"pool": {
"name": "default",
"description": "Default pool",
"id": 0,
"resource_uri": "/MAAS/api/2.0/resourcepool/0/"
},
"used": {
"cores": 0,
"memory": 0,
"local_storage": 0
},
"type": "lxd",
"name": "smoothwall-lxc",
"default_macvlan_mode": null,
"id": 9,
"resource_uri": "/MAAS/api/2.0/vm-hosts/9/"
}
$ maas ben vm-host compose 9 hostname=juju-controller cores=2 memory=4096 storage='bootvol:40(default)' interfaces='eth0:space=admin-space;eth1:space=data-space'
{"cores": ["Ensure this value is less than or equal to 0."], "memory": ["Ensure this value is less than or equal to 0."]}
I thought this answer should be no because of the Q&A on their website:
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: yes - The bridge LXD creates is isolated and not managed by MAAS. If this bridge is used, you would be able to add the LXD VM host and compose virtual machines, but commissioning, deploying, and any other MAAS action which uses the network will fail – so yes is the correct answer here.
Interesting point to say the least and I’ll certainly keep that in mind in the future, but it doesn’t seem relevant to the topic. Perhaps I’m misunderstanding you in how this could effect maas from recognizing cpu and memory resource. For sanity’s sake, I went ahead and redeployed the same machine without creating a new bridge and have LXD use the maas created bridge and i still ran into the same issue where maas wouldn’t let me create the VM. Would you mind explaining what your thought process is on the LXD netwoking setup? Thanks!
wait, the doc already recommends doing lxd init from the command line, not MAAS. how are you initializing LXD from MAAS?
note that there are also some DHCP settings in that section that you need to confirm or change.
I did use LXD from the command line to initialize it on the host.
- Deploy bare-metal machine using Ubuntu 20.04 via MAAS Web UI (not registering as LXD host)
- ssh into the machine and switch to root user
lxd init- Follow the prompts.
- Add the lxd host into maas from the maas kvm tab.
Basically, I was letting lxd create a network bridge when the doc you referenced recommended using an existing bridge that has access to the maas network. (in my case br-ex1)
However, I still don’t see how this can be relevant to this particular issue since maas doesn’t let me create the vm on the host.
Hence:
$ maas ben vm-host compose 9 hostname=juju-controller cores=2 memory=4096 storage='bootvol:40(default)' interfaces='eth0:space=admin-space;eth1:space=data-space' {"cores": ["Ensure this value is less than or equal to 0."], "memory": ["Ensure this value is less than or equal to 0."]}
If it was a networking issue, I would expect maas to let me create the vm but it would fail during commissioning.
Sorry for the confusion and hopefully that clears it up 
1 Like