Temporal database connection error on 3.5.7

Users
1

We upgraded 3.4 to 3.5 a month ago and then moved from an external pg14 server to another pg16 server two weeks ago. We have been using MAAS fairly smoothly before during and after these. As of today, we cannot create libvirt VMs with our usual maas $MAAS_PROFILE pod compose ... - they are created, but do not start, and yield this message a minute or two later.

CREATE_VM='Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))'

temporal-server is crash cycling on the region controller, and this looks relevant (note REGIOND_IP - this has not been used for the database for years):

Sep  2 21:26:29 maas temporal-server[13734]: Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:251): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1037): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:173): sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "REGIOND_IP", user "maas", database "maasdb", no encryption.

I am looking at MaaS 3.5 - Temporal - relation "schema_version" does not exist - #4 by hmohiuddin and Relation "maasserver_routable_pairs" does not exist - #6 by troyanov and wonder if we missed something in the upgrade.

  • Should we run maas-region dbupgrade?
  • Are there upgrade instructions somewhere which include running this?
2

Do you host MAAS on ubuntu?

3

Yes, Ubuntu 22.04, deb packages.

4

probably you did not migrate the temporal schema

5

Can you try sudo maas-region dbupgrade and sudo systemctl restart maas*?

6

Initially I got this:

# maas-region dbupgrade -v 3
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running pre-migrate handlers for application auth
Running pre-migrate handlers for application contenttypes
Running pre-migrate handlers for application sessions
Running pre-migrate handlers for application sites
Running pre-migrate handlers for application maasserver
Running pre-migrate handlers for application metadataserver
Running pre-migrate handlers for application piston3
Running migrations:
  No migrations to apply.
Running post-migrate handlers for application auth
Running post-migrate handlers for application contenttypes
Running post-migrate handlers for application sessions
Running post-migrate handlers for application sites
Running post-migrate handlers for application maasserver
Running post-migrate handlers for application metadataserver
Running post-migrate handlers for application piston3
Running Temporal migrations:
Failed to apply Temporal migrations
2025-09-04T03:01:47.325Z	ERROR	Unable to connect to SQL database.	{"error": "pq: no pg_hba.conf entry for host \"REGIOND_IP\", user \"maas\", database \"maasdb\", no encryption", "logging-call-at": "handler.go:73"}

I couldn’t figure out how to get Temporal to use SSL, so for now we have changed pg_hba.conf from hostssl to host. Success:

...
Running Temporal migrations:
  Applied all migrations.

MAAS works better now, with temporal server logs no longer spewing errors, and VM operation as expected. Thank you. Two items now:

  1. Is it possible to get temporal working with a SSL connection to the database?
  2. Are there instructions somewhere we missed about running this command?
7

Hi @gregoryo2017

MAAS does not support PostgreSQL with SSL enabled out of the box. There is also this topic that refers to PostgreSQL SSL and the Temporal Server configuration FYI: Postgres with SSL works, but disappears when production.yaml is rendered

If your PostgreSQL SSL settings were compatible with how MAAS is using psycopg2 before the introduction of Temporal on 3.5, then a potential patch on the below files could work for you.

As I said, it is not supported so please consider them before applying them at your own risk.

  1. This is needed to run dbupgrade when on SSL. It could be an alternative to what you did on the PostgreSQL side.
diff --git a/src/maasserver/management/commands/dbupgrade.py b/src/maasserver/management/commands/dbupgrade.py
index 18d7156e4..8ab1e7fe8 100644
--- a/src/maasserver/management/commands/dbupgrade.py
+++ b/src/maasserver/management/commands/dbupgrade.py
@@ -149,6 +149,9 @@ class Command(BaseCommand):
                 	get_path("/usr/bin/temporal-sql-tool"),
                 	"--plugin",
                 	"postgres12",
+                	"--tls",
+                	"--tls-ca-file",
+                	"/path/of/ca/with/self/signed",
+                	# or skipping the host verification
+                	# "--tls-disable-host-verification",
                 	"--endpoint",
                 	endpoint,
                 	"--port",
  1. This is needed to operate Temporal Server with PostgreSQL on SSL mode:
diff --git a/src/maasserver/templates/temporal/production.yaml.template b/src/maasserver/templates/temporal/production.yaml.template
index ae8e478a1..4c176518e 100644
--- a/src/maasserver/templates/temporal/production.yaml.template
+++ b/src/maasserver/templates/temporal/production.yaml.template
@@ -23,6 +23,9 @@ persistence:
     	maxConns: 5
     	maxIdleConns: 5
     	maxConnLifetime: "5m"
+    	tls:
+               	enabled: true
+                	caFile: "/path/of/ca/with/self/signed"
+                   # or enableHostVerification: false
 
 	visibility:
   	sql:
@@ -40,6 +43,9 @@ persistence:
     	maxConns: 1
     	maxIdleConns: 1
     	maxConnLifetime: "5m"
+    	tls:
+               	enabled: true
+                	caFile: "/path/of/ca/with/self/signed"
+                   # or enableHostVerification: false
 
 global:
   membership:
8

Links:

9

Thank you for the detailed description and the option to try! I think we will run with plaintext for now and wrap it up in protections such as firewall rules, but I will run it by other relevant people before we decide.

We are operational again and these log messages have stopped, but I am still very interested in not repeating a problem like this again in future. With that in mind, I would like to know if there are pages we should be looking at for upgrade information like this. Should we ordinarily expect to find such information at Release notes and upgrade instructions ?

10

I believe you used a wrong procedure to backup the MAAS database and you missed the temporal schema. This is why I asked you to manually execute the migrations once again to recreate them.

So there are no special upgrade instructions, it’s simply that you did the backup wrong

11

Okay, so it was likely the db migration, not the upgrade, where we went off the path. I am now looking at How to back up MAAS and I don’t see anything particularly special (which is good) there, and nothing stands out as different to our migration procedure. I’ll leave this alone for now, and consider it again next time we are doing any kind of significant database work, or version upgrades.

Thank you all for the assistance here. It is much appreciated.

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.