Hi,
After upgrade to 3.2 (Postgres 12, Raspberry Pi 3b/aarch64, Ubuntu 20.04LTS), MAAS UI shows:
One rack controller is not yet connected to the region. Visit the rack controllers page for more information
In the meantime, rackd.log
continuously logs the following:
2022-12-29 14:19:24 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
2022-12-29 14:19:25 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
2022-12-29 14:19:25 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
2022-12-29 14:19:26 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
2022-12-29 14:19:26 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
2022-12-29 14:19:27 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://192.168.10.4:5240/MAAS)
regiond.log
:
2022-12-29 14:20:25 regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-12-29 14:20:26 regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-12-29 14:20:36 regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-12-29 14:20:37 regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
maas.log
:
(...)
2022-12-29T14:18:43.006990+00:00 maas maas.service_monitor_service: [error] Can't update service statuses, no RPC connection to region.
2022-12-29T14:22:43.012537+00:00 maas maas.service_monitor_service: message repeated 4 times: [ [error] Can't update service statuses, no RPC connection to region.]
2022-12-29T14:23:27.913585+00:00 maas maas.boot_image_download_service: [error] Can't initiate image download, no RPC connection to region.
2022-12-29T14:23:27.914123+00:00 maas maas.dhcp.probe: [error] Can't initiate DHCP probe; no RPC connection to region.
regiond
is running, one process per CPU core:
$ sudo maas status
bind9 RUNNING pid 1381, uptime 1 day, 6:19:57
dhcpd STOPPED Not started
dhcpd6 STOPPED Not started
http RUNNING pid 1522, uptime 1 day, 6:19:08
ntp RUNNING pid 1484, uptime 1 day, 6:19:40
proxy STOPPED Not started
rackd RUNNING pid 1384, uptime 1 day, 6:19:57
regiond RUNNING pid 1385, uptime 1 day, 6:19:57
syslog RUNNING pid 1486, uptime 1 day, 6:19:40
$ ps -ef | grep regiond
root 1385 854 2 Dec28 ? 00:40:05 python3 /snap/maas/25212/bin/regiond
root 1462 1385 10 Dec28 ? 03:18:19 python3 /snap/maas/25212/bin/regiond
root 1464 1385 0 Dec28 ? 00:05:01 python3 /snap/maas/25212/bin/regiond
root 1465 1385 0 Dec28 ? 00:04:38 python3 /snap/maas/25212/bin/regiond
root 1467 1385 0 Dec28 ? 00:05:01 python3 /snap/maas/25212/bin/regiond
Strangely, regiond
is not exposing any endpoints:
$ curl http://192.168.10.4:5240/MAAS/rpc/
{"eventloops": {}}
I enabled debug log on regiond
, in order to investigate the issue and to get the various SQL queries used to populate the response to GET /MAAS/rpc
I understand there is a serie of 2 or 3 queries, which I captured and ran manually:
maasdb=> SELECT "maasserver_regioncontrollerprocess"."id", "maasserver_regioncontrollerprocess"."created", "maasserver_regioncontrollerprocess"."updated", "maasserver_regioncontrollerprocess"."region_id", "maasserver_regioncontrollerprocess"."pid" FROM "maasserver_regioncontrollerprocess" WHERE "maasserver_regioncontrollerprocess"."region_id" IN (1) ORDER BY "maasserver_regioncontrollerprocess"."pid" ASC;
id | created | updated | region_id | pid
-----+-------------------------------+-------------------------------+-----------+------
467 | 2022-12-28 08:04:03.901737+00 | 2022-12-29 14:31:19.93355+00 | 1 | 1462
466 | 2022-12-28 08:04:03.437903+00 | 2022-12-29 14:31:19.736715+00 | 1 | 1464
464 | 2022-12-28 08:04:02.413728+00 | 2022-12-29 14:31:19.23169+00 | 1 | 1465
465 | 2022-12-28 08:04:02.989507+00 | 2022-12-29 14:31:19.418739+00 | 1 | 1467
maasdb=> SELECT "maasserver_regioncontrollerprocessendpoint"."id", "maasserver_regioncontrollerprocessendpoint"."created", "maasserver_regioncontrollerprocessendpoint"."updated", "maasserver_regioncontrollerprocessendpoint"."process_id", "maasserver_regioncontrollerprocessendpoint"."address", "maasserver_regioncontrollerprocessendpoint"."port" FROM "maasserver_regioncontrollerprocessendpoint";
id | created | updated | process_id | address | port
----+---------+---------+------------+---------+------
(0 rows)
It looks like the maasserver_regioncontrollerprocessendpoint
is not being populated with any data, which certainly explain why a call to GET /MAAS/rpc
returns an empty eventloops
dictionary.
Any idea how can I debug further or fix the issue? It looks like the issue is either on Postgres or regiond
side.
Since, I upgraded to MAAS 3.3, Ubuntu 22.04LTS/ Postgres 14 but the issue remains the same.
Thanks!