Temporal database connection error on 3.5.7

Users
1

We upgraded 3.4 to 3.5 a month ago and then moved from an external pg14 server to another pg16 server two weeks ago. We have been using MAAS fairly smoothly before during and after these. As of today, we cannot create libvirt VMs with our usual maas $MAAS_PROFILE pod compose ... - they are created, but do not start, and yield this message a minute or two later.

CREATE_VM='Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))'

temporal-server is crash cycling on the region controller, and this looks relevant (note REGIOND_IP - this has not been used for the database for years):

Sep  2 21:26:29 maas temporal-server[13734]: Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:251): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1037): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:173): sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "REGIOND_IP", user "maas", database "maasdb", no encryption.

I am looking at MaaS 3.5 - Temporal - relation "schema_version" does not exist - #4 by hmohiuddin and Relation "maasserver_routable_pairs" does not exist - #6 by troyanov and wonder if we missed something in the upgrade.

  • Should we run maas-region dbupgrade?
  • Are there upgrade instructions somewhere which include running this?
2

Do you host MAAS on ubuntu?

3

Yes, Ubuntu 22.04, deb packages.

4

probably you did not migrate the temporal schema

5

Can you try sudo maas-region dbupgrade and sudo systemctl restart maas*?

6

Initially I got this:

# maas-region dbupgrade -v 3
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
Running pre-migrate handlers for application maasserver
Running pre-migrate handlers for application metadataserver
Running pre-migrate handlers for application piston3
Running migrations:
  No migrations to apply.
Running post-migrate handlers for application auth
Running post-migrate handlers for application contenttypes
Running post-migrate handlers for application sessions
Running post-migrate handlers for application sites
Running post-migrate handlers for application maasserver
Running post-migrate handlers for application metadataserver
Running post-migrate handlers for application piston3
Running Temporal migrations:
Failed to apply Temporal migrations
2025-09-04T03:01:47.325Z	ERROR	Unable to connect to SQL database.	{"error": "pq: no pg_hba.conf entry for host \"REGIOND_IP\", user \"maas\", database \"maasdb\", no encryption", "logging-call-at": "handler.go:73"}

I couldn’t figure out how to get Temporal to use SSL, so for now we have changed pg_hba.conf from hostssl to host. Success:

...
Running Temporal migrations:
  Applied all migrations.

MAAS works better now, with temporal server logs no longer spewing errors, and VM operation as expected. Thank you. Two items now:

  1. Is it possible to get temporal working with a SSL connection to the database?
  2. Are there instructions somewhere we missed about running this command?