Power settings for multiple machines linked

Users
1

I am at a loss here. I’ve been struggling with recovering a MAAS install that was working fine for months.

maas 3.2.6-12016-g.19812b4da 23947 3.2/stable canonical✓ -

I discovered that multiple machines had the same IPMI IP address A. I ascertained that the first machine actually owned that IP. The second machine’s IP was actually B so I fixed it via the MAAS GUI. The third machine’s IP was actually C, but when I went to fix it, I discovered that the third machine’s IP had also been set to B. So I set it to C and went back to the second machine and it had been set to C now.

How is this possible? I doubled checked that the UIDs were different.

I deleted and re-added the second machine and all seems well now.

FYI, the first machine was NOT affected by my editing of the second and third machines.

2

This sounds like a very strange issue with MAAS managing duplicate IPMI IP addresses. Here are some suggestions on how to respond:

  • You’re right that machines should never end up with conflicting IPMI IPs, even when editing them. The IPs are tied to the unique machine IDs in the database. What the actual heck?
  • I’m not sure how editing one machine could end up changing others. That definitely seems like a bug or database issue.
  • Deleting and re-adding the machines to reset the IPMI parameters was a good workaround.
  • I would suggest filing a bug report detailing what you saw. Include the MAAS logs if possible. The developers can investigate why the IPs were changed incorrectly.
  • For the records that ended up with duplicate IPs, the IPs were probably cached incorrectly somewhere. The devs can probably spot where that happened from logs.
  • In the future, you may want to avoid editing IPMI IPs if possible and set them during add/commissioning instead. But this still seems like an underlying issue.
  • Great catch on noticing the duplicate IPs and tracking down the odd behavior! Filing a bug will help get to the root cause.

Let me know if you need any help with submitting the bug report. The devs should be very interested in this one, since IPMI params getting crossed does seem like a major bug.

3

We have not seen this again since then. If it happens, we will capture more details. At the time, I was desperate to get something done so I didn’t gather up everything I should have.

I would not be against closing this for now and I can re-open (or open a new ticket) if it happens again

1 Like