Updated to 2.6 from 2.5 now UI slow and not usable


#1

Any idea why after the upgrade from 2.5 to 2.6 the UI now is unusable? No logs whatsoever or errors. Postgres every time I go on the machines tab, goes 100% and it takes 2/3 minutes to load ~200 nodes.

Is there any db configuration I should do after the upgrade? The documentation on the upgrade is not existent. If not can anyone point me to what could be the cause of this issue?

MAAS is on a vm, 16 cores and 16gb of ram.


#2

same here, it’s at leasr 10x slower than before


#3

Sorry to hear you are having issues with UI in 2.6. We will work on improvements for the point release.

Could you please switch grouping to none in the machine listing view and report back if the performance returns to previous level?


#4

Doesn’t help, loading machine still very slow, and even after machines fully loaded, it’s still very slow to view single machine


#5

also switching between different view will reload full machine list every time…


#6

Are the machines being loaded in batches, or are you seeing nothing at all until all 200 nodes have been loaded?


#7

It’s useful for us to know how many ~seconds until the table first appears, and then how many seconds until all 200 machines are loaded. Many thanks.


#8

While testing locally I’m seeing 25 machines loading in batches. It took ~30 seconds to load 240 machines and once loaded a UI is responsive.


#9

Unfortunately the problem does not solve for us.

  • The machines are loaded in batches of 25. The first 25 are very, very slow to load (around 2/3 minutes). The rest 175 they load in 30/45 seconds.
  • Removing group does not help, also, once leaving the machine page the grouping resets to the default group by status.
  • Being stupid I did not backup the DB before the upgrade. Is there a way to SAFELY revert to 2.5? We absolutely cannot wait for a point release patch.

#10

I hate to post anywhere just to say “me too” but this is one of those times. Loading 400-some machines takes several minutes now. 2.5 was much faster, and I’m interested in rolling back, but the stable PPA repo only has 2.6 packages on it now.

We’re also having trouble PXE booting on 2.6 due to the changes required for UEFI and our switch firewall ACLs, so this is turning into a mess if I can’t undo it or fix it soon.


#11

Thanks for reporting this and sorry you hit this, it’s something we’re actively looking in to.

I have filed this as https://bugs.launchpad.net/maas/+bug/1835199, please keep ‘me too’ comments to the appropriate part of Launchpad (i.e. https://bugs.launchpad.net/maas/+bug/1835199/+affectsmetoo )

We are looking into a 2.6.1 point release for this.


#12

Thanks for your reply @sparkiegeek . At the same time, a point release in my experience could take weeks. We need a method to revert from 2.6 to 2.5. is that even possible?

If a downgrade is not doable, could you let us know when you have a patch that could at least mitigate the issue?


#13

Simply restore from your backup that you took before upgrading. Recognising that you admit you didn’t make such a backup (eek!) there’s no documented way of downgrading, and I am not familiar with a safe path (for example, there have been database migrations that are one way transformations).

Your best bet is to subscribe to the bug and follow activity there


#14

any updates on those issues? (really hope the fix release could coming out soon

Seems this one only target frontend issue: https://bugs.launchpad.net/maas/+bug/1835199
however we also seen 100% cpu in postgres query as reported in https://bugs.launchpad.net/maas/+bug/1835316

I’ve caught some insane query like:

SELECT “maasserver_node”.“id”, “maasserver_node”.“created”, “maasserver_node”.“updated”, “maasserver_node”.“system_id”, “maasserver_node”.“hardware_uuid”, “maasserver_node”.“hostname”, “maasserver_node”.“description”, “maasserver_node”.“pool_id”, “maasserver_node”.“domain_id”, “maasserver_node”.“address_ttl”, “maasserver_node”.“status”, “maasserver_node”.“previous_status”, “maasserver_node”.“status_expires”, “maasserver_node”.“owner_id”, “maasserver_node”.“bios_boot_method”, “maasserver_node”.“osystem”, “maasserver_node”.“distro_series”, “maasserver_node”.“architecture”, “maasserver_node”.“min_hwe_kernel”, “maasserver_node”.“hwe_kernel”, “maasserver_node”.“node_type”, “maasserver_node”.“parent_id”, “maasserver_node”.“agent_name”, “maasserver_node”.“error_description”, “maasserver_node”.“zone_id”, “maasserver_node”.“cpu_count”, “maasserver_node”.“cpu_speed”, “maasserver_node”.“memory”, “maasserver_node”.“swap_size”, “maasserver_node”.“bmc_id”, “maasserver_node”.“instance_power_parameters”, “maasserver_node”.“power_state”, “maasserver_node”.“power_state_queried”, “maasserver_node”.“power_state_updated”, “maasserver_node”.“last_image_sync”, “maasserver_node”.“token_id”, “maasserver_node”.“error”, “maasserver_node”.“netboot”, “maasserver_node”.“ephemeral_deploy”, “maasserver_node”.“license_key”, “maasserver_node”.“creation_type”, “maasserver_node”.“boot_interface_id”, “maasserver_node”.“boot_cluster_ip”, “maasserver_node”.“boot_disk_id”, “maasserver_node”.“gateway_link_ipv4_id”, “maasserver_node”.“gateway_link_ipv6_id”, “maasserver_node”.“default_user”, “maasserver_node”.“install_rackd”, “maasserver_node”.“install_kvm”, “maasserver_node”.“enable_ssh”, “maasserver_node”.“skip_bmc_config”, “maasserver_node”.“skip_networking”, “maasserver_node”.“skip_storage”, “maasserver_node”.“url”, “maasserver_node”.“dns_process_id”, “maasserver_node”.“managing_process_id”, “maasserver_node”.“current_commissioning_script_set_id”, “maasserver_node”.“current_installation_script_set_id”, “maasserver_node”.“current_testing_script_set_id”, “maasserver_node”.“locked”, (SELECT U2.“description” FROM “maasserver_event” U0 INNER JOIN “maasserver_eventtype” U2 ON (U0.“type_id” = U2.“id”) WHERE (U0.“node_id” = (“maasserver_node”.“id”) AND U2.“level” >= $1) ORDER BY U0.“created” DESC, U0.“id” DESC LIMIT $2) AS “status_event_type_description”, (SELECT U0.“description” FROM “maasserver_event” U0 INNER JOIN “maasserver_eventtype” U2 ON (U0.“type_id” = U2.“id”) WHERE (U0.“node_id” = (“maasserver_node”.“id”) AND U2.“level” >= $3) ORDER BY U0.“created” DESC, U0.“id” DESC LIMIT $4) AS “status_event_description”, “maasserver_domain”.“id”, “maasserver_domain”.“created”, “maasserver_domain”.“updated”, “maasserver_domain”.“name”, “maasserver_domain”.“authoritative”, “maasserver_domain”.“ttl”, “auth_user”.“id”, “auth_user”.“password”, “auth_user”.“last_login”, “auth_user”.“is_superuser”, “auth_user”.“username”, “auth_user”.“first_name”, “auth_user”.“last_name”, “auth_user”.“email”, “auth_user”.“is_staff”, “auth_user”.“is_active”, “auth_user”.“date_joined”, “maasserver_zone”.“id”, “maasserver_zone”.“created”, “maasserver_zone”.“updated”, “maasserver_zone”.“name”, “maasserver_zone”.“description”, “maasserver_bmc”.“id”, “maasserver_bmc”.“created”, “maasserver_bmc”.“updated”, “maasserver_bmc”.“bmc_type”, “maasserver_bmc”.“ip_address_id”, “maasserver_bmc”.“power_type”, “maasserver_bmc”.“power_parameters”, “maasserver_bmc”.“name”, “maasserver_bmc”.“architectures”, “maasserver_bmc”.“capabilities”, “maasserver_bmc”.“cores”, “maasserver_bmc”.“cpu_speed”, “maasserver_bmc”.“memory”, “maasserver_bmc”.“local_storage”, “maasserver_bmc”.“local_disks”, “maasserver_bmc”.“iscsi_storage”, “maasserver_bmc”.“pool_id”, “maasserver_bmc”.“zone_id”, “maasserver_bmc”.“tags”, “maasserver_bmc”.“cpu_over_commit_ratio”, “maasserver_bmc”.“memory_over_commit_ratio”, “maasserver_bmc”.“default_storage_pool_id”, “maasserver_bmc”.“default_macvlan_mode” FROM “maasserver_node” LEFT OUTER JOIN “maasserver_domain” ON (“maasserver_node”.“domain_id” = “maasserver_domain”.“id”) LEFT OUTER JOIN “auth_user” ON (“maasserver_node”.“owner_id” = “auth_user”.“id”) INNER JOIN “maasserver_zone” ON (“maasserver_node”.“zone_id” = “maasserver_zone”.“id”) LEFT OUTER JOIN “maasserver_bmc” ON (“maasserver_node”.“bmc_id” = “maasserver_bmc”.“id”) WHERE (“maasserver_node”.“node_type” = $5 AND “maasserver_node”.“node_type” = $6 AND “maasserver_node”.“id” > $7) ORDER BY “maasserver_node”.“id” ASC LIMIT $8


Db flood by maas event log
#15

confirmed that above query is the problem, after truncate maasserver_event table, cpu issue is gone


#16

Thanks for spotting that. I really hope a point release or a patch could be available soon.

What file did you modify and how for your fix? @hyuwang

Many thanks


#17

pg sql issue can be solved by truncate maasserver_event table (see my other post: Db flood by maas event log ), but there is also frontend performance issue (fixed committed, not release yet), you could try compile from source.