MAAS 3.5.10: machines complete release / power off, but remain stuck in Disk erasing

Users
1

Hi all,

I’m seeing a strange release-state issue on an all-in-one MAAS 3.5.10 install on Jammy. Ours is a fleet of Cisco UCS machines.

A few machines appear to release normally:

  • they enter DISK_ERASING
  • PXE boot the ephemeral environment
  • fetch preseed
  • post metadata status updates
  • MAAS logs: DISK_ERASING -> RELEASING
  • MAAS logs: Turns on netboot for the node

But after that, the machine powers off and MAAS does not always move it to READY.

For example, I had:

  • node-a (efgh34): CIMC showed the machine was off, but machine read still showed power_state: on and status_name: Disk erasing

So this looks less like an actual disk erase problem and more like MAAS getting stuck tracking the final release state.

Has anyone seen this before on 3.5.x?
Also wondering if this is known to be fixed in 3.6 or 3.7.

If helpful, I can share the exact log sequence, but the common pattern is:

  • Release started
  • successful metadata callbacks from the ephemeral environment
  • Releasing node
  • Status transition from DISK_ERASING to RELEASING
  • machine powers off
  • MAAS record stays stale

thanks!

2

I found a better clue from the logs.

On two of the affected machines, MAAS did get as far as:

  • Releasing node
  • Status transition from DISK_ERASING to RELEASING

But right after that, regiond logged:

  • RegionServer connection lost
  • Failed to process status message instantly

and the traceback ends with a Twisted ConnectionLost error.

So this now looks less like just a stale power-state refresh and more like MAAS hitting an internal error during the final part of release.

Has anyone seen this before ?

thanks!

3

I dug a bit more into this and it looks like MAAS is auto-discovering controller IPs from the host itself, including Docker bridge interfaces.

On my all-in-one controller host, rack-controllers read shows not just the real MAAS-facing IPs, but also Docker bridge addresses like 172.17.x.x. On the host, those are real interfaces:

  • docker0
  • a custom bridge for my local exporter container
  • another Docker bridge network

So the RegionServer connection lost messages to <IP> do not appear to be about the managed machine network. They look like MAAS internal traffic using a host-local Docker bridge address.

At this point, the pattern seems to be:

  • machine release progresses normally
  • MAAS logs DISK_ERASING -> RELEASING
  • then regiond logs RegionServer connection lost and Failed to process status message instantly
  • machine powers off, but MAAS state remains stale

So I’m now wondering if the real issue is MAAS auto-detecting and using Docker bridge IPs on the controller host.

Has anyone seen this before, or is there a recommended way to stop MAAS from advertising or using Docker bridge interfaces on a region+rack controller host?

thanks!

4

Would appreciate any insight here.

thanks!