Upgrade Postgresql to 14

[ludww]

Hi guys,

I upgraded Maas to 3.3.4 and Postgresql from 10 to 12, but since it’s deprecated now I would like to upgrade it to 14.
I followed this procedure and it worked like a charm to upgrade to 12:

Upgrading Postgresql to version 12

Upgrade Postgresql from v10 to v12 BACKUP EXISTING DATA Optional but highly recommended step sudo -u postgres pg_dumpall > backup.sql Upgrade the service Install new packages sudo apt-get update sudo apt-get install postgresql-12 postgresql-server-dev-12 Stop the existing server sudo systemctl stop postgresql.service Convert configuration files sudo -u postgres /usr/lib/postgresql/12/bin/pg_upgrade \ --old-datadir=/var/lib/postgresql/10/main \ --new-datadir=/var/lib/postgresql/12/main \ -…

I tried to use the same to go to 14 and I have less success, I can run all the commands and upgrade the DB:

$ sudo -u postgres psql -c "SELECT version();"
                                                                version
----------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 14.8 (Ubuntu 14.8-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, 64-bit
(1 row)

But Maas does not like it, when I load the dashboard I got this:
The server connection failed with the error "Bad Gateway".

It looks like a login issue from regiond service and I can also see errors about RPC on rackd:

$ sudo systemctl status maas-regiond maas-rackd
● maas-regiond.service - MAAS Region Controller
     Loaded: loaded (/lib/systemd/system/maas-regiond.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-06-26 06:36:06 UTC; 1min 24s ago
       Docs: https://maas.io/
   Main PID: 996 (sh)
      Tasks: 3 (limit: 19010)
     Memory: 113.1M
        CPU: 4.396s
     CGroup: /system.slice/maas-regiond.service
             ├─ 996 /bin/sh -c "exec /usr/sbin/regiond 2>&1 |       tee -a \$LOGFILE"
             ├─1082 /usr/bin/python3 /usr/sbin/regiond
             └─1083 tee -a /var/log/maas/regiond.log

Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:   File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 200, in connect
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:     self.connection = self.get_new_connection(conn_params)
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:   File "/usr/lib/python3/dist-packages/django/utils/asyncio.py", line 33, in inner
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:     return func(*args, **kwargs)
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:   File "/usr/lib/python3/dist-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:     connection = Database.connect(**conn_params)
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:   File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 122, in connect
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]:     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]: django.db.utils.OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "maas"
Jun 26 06:37:30 dc1-it-maas-01 sh[1083]: connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for user "maas"

● maas-rackd.service - MAAS Rack Controller
     Loaded: loaded (/lib/systemd/system/maas-rackd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-06-26 06:36:07 UTC; 1min 24s ago
       Docs: https://maas.io/
    Process: 995 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.sock (code=exited, status=0/SUCCESS)
    Process: 1079 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.conf (code=exited, status=0/SUCCESS)
    Process: 1136 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd6.conf (code=exited, status=0/SUCCESS)
   Main PID: 1139 (sh)
      Tasks: 9 (limit: 19010)
     Memory: 193.0M
        CPU: 4.697s
     CGroup: /system.slice/maas-rackd.service
             ├─1139 /bin/sh -c "exec /usr/sbin/rackd 2>&1 | tee -a \$LOGFILE"
             ├─1143 /usr/bin/python3 /usr/sbin/rackd
             ├─1144 tee -a /var/log/maas/rackd.log
             ├─1651 /usr/bin/python3 /usr/lib/maas/maas-common observe-beacons ens160
             ├─1691 sudo -n /usr/lib/maas/beacon-monitor ens160
             └─1692 /usr/bin/tcpdump -Z root --interface ens160 --direction=in --no-promiscuous-mode --packet-buffered --immediate-mode --snapshot-length=16384 -n -w - "(udp dst port 5240) or (vlan and udp dst port 5240)"

Jun 26 06:37:27 dc1-it-maas-01 sh[1144]: 2023-06-26 06:37:27 provisioningserver.rpc.clusterservice: [info] Region is not advertising RPC endpoints. (While requesting RPC info at http://localhost:5240/MAAS)

However the password did not change in regiond.conf
Am I missing something?
Thank you!

[alexsander-souza]

Postgresql 14 has changed the default password hash function from md5 to scram-sha-256, so all passwords must be redefined.

$ sudo -u postgres psql
\password $USER
Enter new password for user "XXX": 
Enter it again: 

alternatively you can revert the change in /etc/postgresql/14/main/pg_hba.conf, but we don’t recommend this as md5 offers weaker security.

[r00ta]

Alternatively, you can use the section Upgrade PostgreSQL of How to upgrade MAAS adjusting it with the versions 12/14 and using snap commands to start/stop maas in case you installed maas with snap.

[ludww]

Thank you, I was not aware of this change, I updated Maas password and it’s ok now.
I am now at Ubuntu 20.04 / Maas 3.3.2 and Postgresql 14.
I can commission and deploy servers but I see some errors in /var/log/maas/regiond.log and I want to make sure everything is fine before continuing the upgrade:

2023-06-27 15:31:56 RegionServer,24,::ffff:192.168.170.108: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:192.168.172.3', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:192.168.170.108', port=34294, flowInfo=0, scopeID=0))
2023-06-27 15:31:56 maasserver.rpc.regionservice: [info] Rack controller 'None' could not be authenticated; dropping connection. Check that /var/lib/maas/secret on the controller contains the correct shared key.
2023-06-27 15:31:58 twisted.internet.protocol.Factory: [info] RegionServer connection established (HOST:IPv6Address(type='TCP', host='::ffff:192.168.172.3', port=5253, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:192.168.170.108', port=42518, flowInfo=0, scopeID=0))
2023-06-27 15:31:59 maasserver.rpc.regionservice: [info] Rack controller 'None' disconnected.

Is it related to the DB upgrade ?
I have a secret in /var/lib/maas/secret but I don’t know what is it for.

Edit: It was fixed while continuing the upgrade so it’s no longer an issue, thanks for you help!