Hello,
I am trying to set up MAAS in the following configuration
I have not set up the rack controller yet. I am on the Region controller UI, trying to set it up through the first-time log in prompts. The images never download even though the region controller has internet access.
Even setting up the rack controller and connecting it to the region controller doesn’t change anything (as seen below):
If I add any image, it goes to the ‘Queued for download’ state and if I try to delete it, I get the following error:
I am using the MAAS snap through the web UI on a Ubuntu 24.04 server (for both region and rack controllers) with the root partition having almost 80 GB free on both controllers. I can ping google.com and maas.io from both controllers.
Both controllers have the same MAAS version:
$ snap list maas
Name Version Rev Tracking Publisher Notes
maas 3.5.3-16341-g.7adb035d6 38185 3.5/stable canonical✓ -
Logs:
I ran journalctl -u snap.maas.pebble.service to get the following.
Feb 25 19:57:41 region-controller maas.pebble[3563]: 2025-02-25T19:57:41.297Z [pebble] Change 75 task (Start service "temporal") failed: cannot start service: exited quickly with code 1
Feb 25 19:57:42 region-controller maas-log[4197]: maas.service_monitor: [error] Service 'temporal' failed to start: Pebble change 75 failed with an error: cannot perform the following tasks:#012- Start service "temporal" (cannot start service: exited quickly with code 1)
Feb 25 19:57:42 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: Task exception was never retrieved
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: future: <Task finished name='Task-2' coro=<Worker.run() done, defined at /snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py:99> exception=RuntimeError('Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))')>
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: Traceback (most recent call last):
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 100, in run
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: self._client = self._client or await get_client_async()
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 263, in wrapped
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: raise e
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 259, in wrapped
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: return await fn(*args, **kwargs)
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 45, in get_client_async
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: return await Client.connect(
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/client.py", line 148, in connect
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: await temporalio.service.ServiceClient.connect(connect_config),
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 158, in connect
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: return await _BridgeServiceClient.connect(config)
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 685, in connect
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: await client._connected_client()
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 698, in _connected_client
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: self._bridge_client = await temporalio.bridge.client.Client.connect(
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/bridge/client.py", line 78, in connect
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: await temporalio.bridge.temporal_sdk_bridge.connect_client(
Feb 25 19:57:44 region-controller maas-temporal-worker[23657]: RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))
Feb 25 19:57:45 region-controller maas.pebble[3563]: 2025-02-25T19:57:45.120Z [pebble] Service "temporal-worker" stopped unexpectedly with code 0
Feb 25 19:57:45 region-controller maas.pebble[3563]: 2025-02-25T19:57:45.120Z [pebble] Service "temporal-worker" on-success action is "restart", waiting ~500ms before restart (backoff 1)
Feb 25 19:57:45 region-controller maas.pebble[3563]: 2025-02-25T19:57:45.641Z [pebble] Service "temporal-worker" starting: sh -c "exec systemd-cat -t maas-temporal-worker $SNAP/bin/run-temporal-worker"
Feb 25 19:57:45 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:57:49 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:57:52 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:57:55 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:57:58 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:01 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:04 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:07 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:10 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=bind9 152.771µs 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=ntp 122.403µs 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=http 36.299µs 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=syslog 76.485µs 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=temporal-worker 74.752µs 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.141Z [pebble] GET /v1/services?names=temporal 121.891µs 200
Feb 25 19:58:11 region-controller maas-log[4197]: maas.service_monitor: [info] Service 'maas-temporal' is not on, it will be started.
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.163Z [pebble] POST /v1/services 6.841646ms 202
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.163Z [pebble] GET /v1/services?names=proxy 5.978325ms 200
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.169Z [pebble] Service "temporal" starting: sh -c "exec systemd-cat -t maas-temporal $SNAP/bin/run-temporal"
Feb 25 19:58:11 region-controller maas-temporal[24288]: 2025/02/25 19:58:11 Loading config; env=production,zone=,configDir=/var/snap/maas/38185/temporal
Feb 25 19:58:11 region-controller maas-temporal[24288]: 2025/02/25 19:58:11 Loading config files=[/var/snap/maas/38185/temporal/production.yaml]
Feb 25 19:58:11 region-controller maas-temporal[24288]: {"level":"error","ts":"2025-02-25T19:58:11.261Z","msg":"error returned","component":"fx","name":"go.temporal.io/server/common/membership/ringpop.provideHostInfoProvider()","kind":"provide","module":"","error":"received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","logging-call-at":"fx.go:1122","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1122\ngo.uber.org/fx.(*module).provide.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:160\ngo.uber.org/dig.(*constructorNode).Call.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:161\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:182\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:377\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.org/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:11 region-controller maas-temporal[24288]: {"level":"error","ts":"2025-02-25T19:58:11.261Z","msg":"invoke failed","component":"fx","error":"could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","stack":"go.temporal.io/server/common/deadlock.init\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6506\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:233\n","function":"go.temporal.io/server/common/deadlock.glob..func1()","module":"","logging-call-at":"fx.go:1139","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1139\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:280\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:377\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.org/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:11 region-controller maas-temporal[24288]: {"level":"error","ts":"2025-02-25T19:58:11.262Z","msg":"error returned","component":"fx","name":"go.temporal.io/server/temporal.HistoryServiceProvider()","kind":"provide","module":"","error":"received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","logging-call-at":"fx.go:1122","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1122\ngo.uber.org/fx.(*module).provide.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:160\ngo.uber.org/dig.(*constructorNode).Call.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:161\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:182\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:11 region-controller maas-temporal[24288]: {"level":"error","ts":"2025-02-25T19:58:11.262Z","msg":"invoke failed","component":"fx","error":"could not build arguments for function \"go.temporal.io/server/temporal\".ServerLifetimeHooks (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881): failed to build *temporal.ServerImpl: could not build arguments for function \"go.temporal.io/server/temporal\".NewServerFxImpl (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64): could not build value group *temporal.ServicesMetadata[group=\"services\"]: received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"could not build arguments for function \"go.temporal.io/server/temporal\".ServerLifetimeHooks\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881:\nfailed to build *temporal.ServerImpl:\ncould not build arguments for function \"go.temporal.io/server/temporal\".NewServerFxImpl\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64:\ncould not build value group *temporal.ServicesMetadata[group=\"services\"]:\nreceived non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","stack":"go.temporal.io/server/temporal.init\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:154\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6506\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:233\n","function":"go.temporal.io/server/temporal.ServerLifetimeHooks()","module":"","logging-call-at":"fx.go:1139","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1139\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:280\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:11 region-controller maas-temporal[24288]: Unable to create server. Error: could not build arguments for function "go.temporal.io/server/temporal".ServerLifetimeHooks (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881): failed to build *temporal.ServerImpl: could not build arguments for function "go.temporal.io/server/temporal".NewServerFxImpl (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64): could not build value group *temporal.ServicesMetadata[group="services"]: received non-nil error from function "go.temporal.io/server/temporal".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function "go.temporal.io/server/common/deadlock".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function "go.temporal.io/server/common/deadlock".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group="deadlockDetectorRoots"]: could not build arguments for function "reflect".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function "go.temporal.io/server/service/history/shard".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function "go.temporal.io/server/service/history/shard".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function "go.temporal.io/server/common/membership/ringpop".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::]).
Feb 25 19:58:11 region-controller maas.pebble[3563]: 2025-02-25T19:58:11.272Z [pebble] Change 76 task (Start service "temporal") failed: cannot start service: exited quickly with code 1
Feb 25 19:58:12 region-controller maas-log[4197]: maas.service_monitor: [error] Service 'temporal' failed to start: Pebble change 76 failed with an error: cannot perform the following tasks:#012- Start service "temporal" (cannot start service: exited quickly with code 1)
Feb 25 19:58:13 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: Task exception was never retrieved
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: future: <Task finished name='Task-2' coro=<Worker.run() done, defined at /snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py:99> exception=RuntimeError('Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))')>
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: Traceback (most recent call last):
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 100, in run
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: self._client = self._client or await get_client_async()
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 263, in wrapped
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: raise e
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 259, in wrapped
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: return await fn(*args, **kwargs)
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 45, in get_client_async
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: return await Client.connect(
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/client.py", line 148, in connect
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: await temporalio.service.ServiceClient.connect(connect_config),
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 158, in connect
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: return await _BridgeServiceClient.connect(config)
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 685, in connect
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: await client._connected_client()
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 698, in _connected_client
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: self._bridge_client = await temporalio.bridge.client.Client.connect(
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/bridge/client.py", line 78, in connect
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: await temporalio.bridge.temporal_sdk_bridge.connect_client(
Feb 25 19:58:16 region-controller maas-temporal-worker[24028]: RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))
Feb 25 19:58:16 region-controller maas.pebble[3563]: 2025-02-25T19:58:16.571Z [pebble] Service "temporal-worker" stopped unexpectedly with code 0
Feb 25 19:58:16 region-controller maas.pebble[3563]: 2025-02-25T19:58:16.571Z [pebble] Service "temporal-worker" on-success action is "restart", waiting ~500ms before restart (backoff 1)
Feb 25 19:58:16 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:17 region-controller maas.pebble[3563]: 2025-02-25T19:58:17.100Z [pebble] Service "temporal-worker" starting: sh -c "exec systemd-cat -t maas-temporal-worker $SNAP/bin/run-temporal-worker"
Feb 25 19:58:19 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:21 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 0.0.0.0 moved from 00:00:1a:1f:1f:34 to cc:28:aa:a6:8e:08
Feb 25 19:58:23 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:26 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:29 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:32 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:35 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:38 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.139Z [pebble] GET /v1/services?names=bind9 118.365µs 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.139Z [pebble] GET /v1/services?names=ntp 96.834µs 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.140Z [pebble] GET /v1/services?names=syslog 52.48µs 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.140Z [pebble] GET /v1/services?names=http 46.899µs 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.140Z [pebble] GET /v1/services?names=temporal-worker 34.235µs 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.140Z [pebble] GET /v1/services?names=temporal 172.618µs 200
Feb 25 19:58:41 region-controller maas-log[4197]: maas.service_monitor: [info] Service 'maas-temporal' is not on, it will be started.
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.158Z [pebble] POST /v1/services 10.259292ms 202
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.165Z [pebble] GET /v1/services?names=proxy 13.208279ms 200
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.165Z [pebble] Service "temporal" starting: sh -c "exec systemd-cat -t maas-temporal $SNAP/bin/run-temporal"
Feb 25 19:58:41 region-controller maas-temporal[24701]: 2025/02/25 19:58:41 Loading config; env=production,zone=,configDir=/var/snap/maas/38185/temporal
Feb 25 19:58:41 region-controller maas-temporal[24701]: 2025/02/25 19:58:41 Loading config files=[/var/snap/maas/38185/temporal/production.yaml]
Feb 25 19:58:41 region-controller maas-temporal[24701]: {"level":"error","ts":"2025-02-25T19:58:41.272Z","msg":"error returned","component":"fx","name":"go.temporal.io/server/common/membership/ringpop.provideHostInfoProvider()","kind":"provide","module":"","error":"received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","logging-call-at":"fx.go:1122","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1122\ngo.uber.org/fx.(*module).provide.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:160\ngo.uber.org/dig.(*constructorNode).Call.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:161\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:182\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:377\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.org/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:41 region-controller maas-temporal[24701]: {"level":"error","ts":"2025-02-25T19:58:41.273Z","msg":"invoke failed","component":"fx","error":"could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","stack":"go.temporal.io/server/common/deadlock.init\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6506\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:233\n","function":"go.temporal.io/server/common/deadlock.glob..func1()","module":"","logging-call-at":"fx.go:1139","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1139\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:280\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:377\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.org/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:41 region-controller maas-temporal[24701]: {"level":"error","ts":"2025-02-25T19:58:41.274Z","msg":"error returned","component":"fx","name":"go.temporal.io/server/temporal.HistoryServiceProvider()","kind":"provide","module":"","error":"received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","logging-call-at":"fx.go:1122","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1122\ngo.uber.org/fx.(*module).provide.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:160\ngo.uber.org/dig.(*constructorNode).Call.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:161\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:182\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:41 region-controller maas-temporal[24701]: {"level":"error","ts":"2025-02-25T19:58:41.274Z","msg":"invoke failed","component":"fx","error":"could not build arguments for function \"go.temporal.io/server/temporal\".ServerLifetimeHooks (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881): failed to build *temporal.ServerImpl: could not build arguments for function \"go.temporal.io/server/temporal\".NewServerFxImpl (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64): could not build value group *temporal.ServicesMetadata[group=\"services\"]: received non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group=\"deadlockDetectorRoots\"]: could not build arguments for function \"reflect\".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::])","errorVerbose":"could not build arguments for function \"go.temporal.io/server/temporal\".ServerLifetimeHooks\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881:\nfailed to build *temporal.ServerImpl:\ncould not build arguments for function \"go.temporal.io/server/temporal\".NewServerFxImpl\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64:\ncould not build value group *temporal.ServicesMetadata[group=\"services\"]:\nreceived non-nil error from function \"go.temporal.io/server/temporal\".HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35:\nfailed to build *deadlock.deadlockDetector:\ncould not build arguments for function \"go.temporal.io/server/common/deadlock\".NewDeadlockDetector\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82:\ncould not build value group common.Pingable[group=\"deadlockDetectorRoots\"]:\ncould not build arguments for function \"reflect\".makeFuncStub\n\t/usr/lib/go-1.20/src/reflect/asm_amd64.s:28:\nfailed to build shard.Controller:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".glob..func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36:\nfailed to build *shard.ControllerImpl:\ncould not build arguments for function \"go.temporal.io/server/service/history/shard\".ControllerProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91:\nfailed to build membership.HostInfoProvider:\nreceived non-nil error from function \"go.temporal.io/server/common/membership/ringpop\".provideHostInfoProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55:\nbroadcastAddress required when listening on all interfaces (0.0.0.0/[::])","stack":"go.temporal.io/server/temporal.init\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:154\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6506\nruntime.doInit\n\t/usr/lib/go-1.20/src/runtime/proc.go:6483\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:233\n","function":"go.temporal.io/server/temporal.ServerLifetimeHooks()","module":"","logging-call-at":"fx.go:1139","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:156\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:1139\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:280\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/command.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}
Feb 25 19:58:41 region-controller maas-temporal[24701]: Unable to create server. Error: could not build arguments for function "go.temporal.io/server/temporal".ServerLifetimeHooks (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:881): failed to build *temporal.ServerImpl: could not build arguments for function "go.temporal.io/server/temporal".NewServerFxImpl (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/server_impl.go:64): could not build value group *temporal.ServicesMetadata[group="services"]: received non-nil error from function "go.temporal.io/server/temporal".HistoryServiceProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:367): could not build arguments for function "go.temporal.io/server/common/deadlock".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/fx.go:35): failed to build *deadlock.deadlockDetector: could not build arguments for function "go.temporal.io/server/common/deadlock".NewDeadlockDetector (/build/temporal-8dYnj9/temporal-1.22.5/src/common/deadlock/deadlock.go:82): could not build value group common.Pingable[group="deadlockDetectorRoots"]: could not build arguments for function "reflect".makeFuncStub (/usr/lib/go-1.20/src/reflect/asm_amd64.s:28): failed to build shard.Controller: could not build arguments for function "go.temporal.io/server/service/history/shard".glob..func1 (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/fx.go:36): failed to build *shard.ControllerImpl: could not build arguments for function "go.temporal.io/server/service/history/shard".ControllerProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/service/history/shard/controller_impl.go:91): failed to build membership.HostInfoProvider: received non-nil error from function "go.temporal.io/server/common/membership/ringpop".provideHostInfoProvider (/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:55): broadcastAddress required when listening on all interfaces (0.0.0.0/[::]).
Feb 25 19:58:41 region-controller maas.pebble[3563]: 2025-02-25T19:58:41.286Z [pebble] Change 77 task (Start service "temporal") failed: cannot start service: exited quickly with code 1
Feb 25 19:58:41 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:42 region-controller maas-log[4197]: maas.service_monitor: [error] Service 'temporal' failed to start: Pebble change 77 failed with an error: cannot perform the following tasks:#012- Start service "temporal" (cannot start service: exited quickly with code 1)
Feb 25 19:58:44 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:47 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: Task exception was never retrieved
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: future: <Task finished name='Task-2' coro=<Worker.run() done, defined at /snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py:99> exception=RuntimeError('Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))')>
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: Traceback (most recent call last):
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 100, in run
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: self._client = self._client or await get_client_async()
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 263, in wrapped
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: raise e
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/utils/asynchronous.py", line 259, in wrapped
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: return await fn(*args, **kwargs)
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/lib/python3.10/site-packages/maasserver/workflow/worker/worker.py", line 45, in get_client_async
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: return await Client.connect(
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/client.py", line 148, in connect
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: await temporalio.service.ServiceClient.connect(connect_config),
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 158, in connect
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: return await _BridgeServiceClient.connect(config)
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 685, in connect
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: await client._connected_client()
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/service.py", line 698, in _connected_client
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: self._bridge_client = await temporalio.bridge.client.Client.connect(
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: File "/snap/maas/38185/usr/lib/python3/dist-packages/temporalio/bridge/client.py", line 78, in connect
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: await temporalio.bridge.temporal_sdk_bridge.connect_client(
Feb 25 19:58:47 region-controller maas-temporal-worker[24363]: RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))
Feb 25 19:58:48 region-controller maas.pebble[3563]: 2025-02-25T19:58:48.055Z [pebble] Service "temporal-worker" stopped unexpectedly with code 0
Feb 25 19:58:48 region-controller maas.pebble[3563]: 2025-02-25T19:58:48.055Z [pebble] Service "temporal-worker" on-success action is "restart", waiting ~500ms before restart (backoff 1)
Feb 25 19:58:48 region-controller maas.pebble[3563]: 2025-02-25T19:58:48.594Z [pebble] Service "temporal-worker" starting: sh -c "exec systemd-cat -t maas-temporal-worker $SNAP/bin/run-temporal-worker"
Feb 25 19:58:50 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:f0 to c0:18:50:cb:18:ef
Feb 25 19:58:53 region-controller maas-log[4197]: maas.neighbour: [info] enp65s0 (physical) on region-controller: IP address 10.228.21.168 moved from c0:18:50:cb:18:ef to c0:18:50:cb:18:f0
Please let me know if you need anything else.
Thank you!