Hello! My team runs a TLS-enabled MaaS instance that has an intriguing problem: it can download Ubuntu images, or it can start the DHCP server, but apparently not both at the same time. I’ve been able to reproduce this behavior with a fresh (and what seems to me to be a) minimal configuration with a single region+rack controllre, with the variable being the MaaS URL provided during initial configuration.
We are using a Let’s Encrypt-provided certificate to enable native TLS in our MaaS environment.
To configure the reproduction environment:
- Install Ubuntu Server
- Install the MaaS 3.7.2 snap
- Install PostgreSQL using
apt - Initialize the MaaS PostgreSQL database
- Configure the node as a region+rack controller using the specified MaaS URL (more on that detail later)
- Configure MaaS-native TLS using
maas config-tls - Create the admin user
- Through the web UI, finish configuring the region
When the MaaS URL specifies a non-TLS-enabled endpoint (i.e., http://regionandrack.mydomain.com:5240/MAAS), the controller downloads Ubuntu images with impressive speed. However, once I enable DHCP on a subnet, the DHCP server never starts:
$ sudo maas status
Service Startup Current Since
agent disabled active today at 18:16 UTC
apiserver enabled active today at 18:15 UTC
bind9 disabled active today at 18:15 UTC
dhcpd disabled inactive -
dhcpd6 disabled inactive -
http disabled active today at 18:15 UTC
ntp disabled active today at 18:15 UTC
proxy disabled inactive today at 18:18 UTC
rackd enabled active today at 18:15 UTC
regiond enabled active today at 18:15 UTC
syslog disabled active today at 18:16 UTC
temporal disabled active today at 18:15 UTC
temporal-worker disabled active today at 18:15 UTC
Also, the DHCP configuration files are zero-length:
$ ls -alh /var/snap/maas/common/maas/dhcp*
-rw-r----- 1 root root 0 Apr 9 18:19 /var/snap/maas/common/maas/dhcpd6.conf
-rw-r----- 1 root root 0 Apr 9 18:19 /var/snap/maas/common/maas/dhcpd6-interfaces
-rw-r----- 1 root root 0 Apr 9 18:19 /var/snap/maas/common/maas/dhcpd.conf
-rw-r----- 1 root root 0 Apr 9 18:19 /var/snap/maas/common/maas/dhcpd-interfaces
srw-rw-rw- 1 root root 0 Apr 9 18:16 /var/snap/maas/common/maas/dhcpd.sock
When the MaaS URL specifies a TLS-enabled endpoint (i.e., https://regionandrack.mydomain.com/MAAS), the controller has no problem populating the DHCP configuration files (with a note that this environment does not have any DHCPv6-enabled interfaces):
$ ls -alh /var/snap/maas/common/maas/dhcp*
-rw-r----- 1 root root 1.1K Apr 9 20:47 /var/snap/maas/common/maas/dhcpd6.conf
-rw-r----- 1 root root 0 Apr 9 20:47 /var/snap/maas/common/maas/dhcpd6-interfaces
-rw-r----- 1 root root 5.2K Apr 9 20:47 /var/snap/maas/common/maas/dhcpd.conf
-rw-r----- 1 root root 6 Apr 9 20:47 /var/snap/maas/common/maas/dhcpd-interfaces
srw-rw-rw- 1 root root 0 Apr 9 20:47 /var/snap/maas/common/maas/dhcpd.sock
And the DHCP server starts with ease:
$ sudo maas status
Service Startup Current Since
agent disabled active today at 20:49 UTC
apiserver enabled active today at 20:49 UTC
bind9 disabled active today at 20:49 UTC
dhcpd disabled active today at 20:49 UTC
dhcpd6 disabled inactive -
http disabled active today at 20:49 UTC
ntp disabled active today at 20:49 UTC
proxy disabled active today at 20:50 UTC
rackd enabled active today at 20:49 UTC
regiond enabled active today at 20:49 UTC
syslog disabled active today at 20:49 UTC
temporal disabled active today at 20:49 UTC
temporal-worker disabled active today at 20:49 UTC
However, the controller indicates that Ubuntu images selected during the UI-based region configuration process are queued for download - but they never download:
Also, the maas-temporal-worker persistently complains about an SSL failure, tweaked to hopefully render into a format that’s more easily read by us humans:
$ journalctl -u snap.maas.pebble.service -t maas-temporal-worker --grep "Completing activity as fai
led" | tail -n 1 | sed 's/^[^{]*//g' | jq . | sed 's/^"//;s/"$//;s/\\n/\n/g;s/\\"/"/g'
{
"message": "Completing activity as failed ({'activity_id': '1', 'activity_type': 'get-bootresourcefile-endpoints', 'attempt': 7, 'namespac
e': 'default', 'task_queue': 'region', 'workflow_id': 'sync-boot-resources:streams', 'workflow_run_id': '9c427897-2568-4e6e-8da6-74f43b97ec0
2', 'workflow_type': 'sync-bootresources'})",
"exc_info": "Traceback (most recent call last):
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
result = func(*args)
^^^^^^^^^^^
File "/usr/lib/python3.12/ssl.py", line 917, in do_handshake
self._sslobj.do_handshake()
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1000)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/_core/_tasks.py", line 115, in fail_after
yield cancel_scope
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_backends/anyio.py", line 69, in start_tls
ssl_stream = await anyio.streams.tls.TLSStream.wrap(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 132, in wrap
await wrapper._call_sslobject_method(ssl_object.do_handshake)
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 147, in _call_sslobject_method
data = await self.transport_stream.receive()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/_backends/_asyncio.py", line 1245, in receive
await self._wait_until_readable(loop)
asyncio.exceptions.CancelledError: Cancelled by cancel scope 726c8c2d2db0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_backends/anyio.py", line 78, in start_tls
raise exc
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_backends/anyio.py", line 68, in start_tls
with anyio.fail_after(timeout):
File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/snap/maas/x1/usr/lib/python3/dist-packages/anyio/_core/_tasks.py", line 118, in fail_after
raise TimeoutError
TimeoutError
The above exception was the direct cause of the following exception:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
yield
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 371, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_async/connection.py", line 156, in _connect
stream = await stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_backends/anyio.py", line 66, in start_tls
with map_exceptions(exc_map):
File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last): 20:54:23 [18/1884]
File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 453, in _run_activity
result = await impl.execute_activity(input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/temporalio/worker/_activity.py", line 711, in execute_activity
return await input.fn(*input.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/lib/python3.12/site-packages/maastemporalworker/workflow/utils.py", line 32, in wrapper
res = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/lib/python3.12/site-packages/maastemporalworker/workflow/bootresource.py", line 189, in get_bootresourcefile_endpoints
regions = await self.apiclient.request_async("GET", url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/lib/python3.12/site-packages/maastemporalworker/workflow/api_client.py", line 62, in request_async
response = await self.unix_client.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_client.py", line 1559, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_client.py", line 1646, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_client.py", line 1674, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_client.py", line 1711, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_client.py", line 1748, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 370, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/snap/maas/x1/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout",
"taskName": "Task-87",
"temporal_activity": {
"activity_id": "1",
"activity_type": "get-bootresourcefile-endpoints",
"attempt": 7,
"namespace": "default",
"task_queue": "region",
"workflow_id": "sync-boot-resources:streams",
"workflow_run_id": "9c427897-2568-4e6e-8da6-74f43b97ec02",
"workflow_type": "sync-bootresources
},
"logger": "temporalio.activity:493",
"level": "WARNING",
"thread": "MainThread:125810733500224",
"timestamp": "2026-04-09T20:53:24.140445Z
}
So it would seem that if we configure TLS, we truly can have either image downloads or a configured and running dhcpd but not both…
I’m very open to the possibility that I’m missing something rather obvious. I’m happy to share any additional non-sensitive configuration details or logs if they would be of any help.
Thank you in advance for your assistance!