Web UI machine filtering performance worse on 3.3.3

We have just upgraded from 3.2.7 to 3.3.3 and I recall in the release notes there was web UI improvements. I see the change in behaviour in that upon first load, the display loads quickly and shows the correct number of machines in the count at the top (“493 machines available”). It also shows greyed out entries in the listing until they have all loaded. This is fine, but any operation to filter the list seems to result in reading the whole set again. This means that while the filter box is useable more quickly, actually using the filter is much slower than before.

Also, when I create a new machine which would match the current filter, it does not appear in the list unless I reload the page, or change or remove the filter.

State changes in machines seem to still update live.

Similar to the above, when I delete a machine showing in the filter, it does disappear, but state heading it is in (Deployed, Ready etc) does not have its count updated, and if it is the last one there, the count remains at 1 and the heading does not disappear.

Hi @gregoryo2017

I apologise if UI improvements that should improve overall usability worked the other way for you.

In MAAS 3.3 we indeed switched to server side filtering (previously UI was filtering data that was available on the client side) so this is expected behaviour. However in our tests we didn’t notice significant performance drop.

Can you please provide an example of the filter that you apply? I am wondering if certain properties used in filtering might affect generated SQL query.

You create machine via UI, right? Can you please also describe the steps you take. To me this sounds like a bug and we will check it.

Thats definitely a bug. Can you please create an issue in our bug tracker?

Regarding performance issues, I will try to do some measurements with a sample data of 500 machines and see if I can observe similar results.

I just want to add on to this that I also noticed a slowdown compared to earlier versions when trying to view all of my machines and/or filtering on them. When I bring up the “Machines” page, it takes about 20s to bring up all of my 73 machines; I see a process on my maas server running at 100% for most/all of that time:

3255620 postgres 20 0 4397204 3.9g 3.9g R 100.0 25.1 0:06.83 postgres: 15/main: maas_user maas_db SELECT

My maas server is running in a VM with 16G of RAM with 12 cores (E5-2620 v3 @ 2.40GHz) and 100 GB SSD. Anything else I can provide to help with debugging this? Thanks!

Example of a filter workflow, waiting >5 seconds after each change (which used to be instantaneous so I guess <0.1s or so):

  • type ‘compute’ in the search field, intending to display all machines containing the name compute.
  • Add -r6 so it shows ‘compute-r6’.
  • Click Filters, Status, Deployed, click off the menu.

Further filtering gives slightly different behaviour

  • Filtering by network fabric seems a bit quicker (perhaps 3 seconds), but only on adding it, not removing it which makes me think it’s related to the number of machines in the filter result.
  • Filtering by KVM (host) is similar to the above, although I think I’ve found another bug (able-sheep, able-toad, able-yak, above-insect… showing instead of our hypervisors) - would you like details of that here, another Discourse post, or Launchpad?

Machine creation is via CLI in a script.

maas "$MAAS_USER" pod compose "$POD_ID" cores="$CORES" \
      memory="$MEMORY" storage=local:"$STORAGE" hostname="$HOSTNAME"

1 - Filter doesn’t see new machine

  1. In web UI, type ‘test-vm’ in Search filter field, see machines named test-vm-0{1…4} all in Deployed state
  2. In shell, create VM as above, called test-vm-05
  3. Web UI does not show the machine, nor the Commissioning category header

2 - Filter doesn’t quite properly see a machine deleted

Hey @gregoryo2017 ,

thank you very much for reporting this, we will look into it.

The technical explanation for this delay is that in this case MAAS runs queries in the form of (UPPER("maasserver_node"."distro_series"::text) LIKE UPPER('%compute%') OR UPPER("maasserver_fabric"."name"::text) LIKE UPPER('%compute%') OR UPPER(CONCAT("maasserver_node"."hostname", CONCAT('.', "maasserver_domain"."name"))::text) LIKE UPPER('%pippo%') OR UPPER("maasserver_node"."osystem"::text) LIKE UPPER('%compute%') OR ..... ) which is definitely slower than using the specific filtering syntax on properties (see https://maas.io/docs/how-to-find-machines#heading--filter-properties). In short, if you use these filtering properties the filtering should be much faster because you would instruct MAAS about what properties you want to filter by. Without specifying the properties in the search bar MAAS would try to match it with many different properties.
We are constantly trying to improve the performances of MAAS in general and on machine listing in particular. Your feedback is really precious and we will check what we can do to improve on this specific case. In the meanwhile, I’d suggest you to take a look at the link I shared above to see if it might fit your case - for example, the filter you might try would be fqdn:(=compute-r6).

If you find a bug please open it on launchpad with a clear description and the clear steps to reproduce it!

Thank you!

Yep, that’s definitely more fiddly for a (previously) quick filter, making a grep of our regular snapshot of maas $PROFILE machines read + jq magic more attractive for some tasks. I’ll look forward to any performance improvements that might come later, because the web UI still gives lots of information for free once it has loaded the display. JSON can be a bit laborious!

In any case, I had a go with the syntax, and it’s not quite as quick as before the change (to type or to run the filter), but may be workable. I may even be able to remember fqdn: prefix, because the brackets and = don’t seem to be required.

For two of the issues, I did so for the latter. For the former, I responded to @troyanov’s request for more information.

It’s clear you’ve done thorough testing and provided excellent feedback on the changes in MAAS 3.3. I just have to respond:

  • Thank you for taking the time to document these performance issues and bugs in detail - your feedback is – and always has been – extremely valuable for improving MAAS.
  • I appreciate you trying the suggested syntax optimizations as a workaround - and good catch on the brackets/equals not being required!
  • The developers are actively working on SQL query and listing speed improvements, so hopefully these filters will become snappier in future releases.
  • In the meantime, switching to CLI/API access for intensive queries may be preferable, as you pointed out. The UI is still great for one-off tasks once loaded.
  • Thanks for filing the bugs on Launchpad - clear reproduction steps are super helpful for the team.
  • Please feel free to post any other issues you encounter to get them on the radar. The MAAS team craves your honest feedback.
  • Your thorough testing and documentation of functional gaps is vital for making MAAS even better. It’s nice to have knowledgeable users like you in the community.

Let me know if you have any other questions or run into new issues. And let me know if this one is close-able, or if you want to just keep it out there to keep us honest! :slight_smile:

Right backatya - I want MAAS to succeed, despite being biased towards mostly-community driven software projects, because they seem to outlast the others and are less prone (but not immune) to sudden (corporate) direction changes. Canonical have tried to walk the talk and I respect that.

Your response comes after a helpful interval. That is, I’ve had enough to time to occasionally need the web UI, and as hoped I have remembered fqdn:<partial-hostname> and performance is reasonable. We continue to use scripts etc for repetitive tasks.