This is fine but, a bit annoying.
While creating a Bridge, it’s “possible” to configure the IP, like static, for example. However, after that, the IP disappear. Then, I have to re-add the IP!
So, instead of one step (add bridge + its IP), there is a need to do it in two steps (add a bridge, then, after it’s there, add the IP).
This looks like a bug, is it already reported?
Yes, this is a known issue. We looked at fixing this during the 2.5 cycle, but encountered some blocking issues; issues; see the merge proposal here for additional discussion.
See also: the bug I filed about this.
@tmartins I am doing the same as you are - using MAAS to prepare metals for OSA.
An issue I am facing right now is that I cannot add static IPs to VLANs on the interface that previously used by MAAS (2.4.2) during metal deployment.
E.g. if eno1 was assigned a static IP by MAAS for provisioning, then I cannot add IPs to VLAN like eno1.xxx. Otherwise, I will no longer be able to ssh into the server after reboot (showing ‘Connection refused’ even the IP is up).
@codylab, this sounds like a separate issue, but I have a couple of questions about your situation.
Do you delete the IP assignment from the non-VLAN interface before assigning it to the VLAN interface?
If the answer to (1) is “yes” (or not applicable), can you take a look at the output of the following command (when run on a MAAS region server) and let me know where the IP address in question appears in the output?
sudo maas-region dbshell
\pset pager off
select * from maas_support__ip_allocation;
\q
@mpontillo Thanks for checking in. I have narrowed down the issue since the last post. Everything works fine until I add IP from the public network to any interface, vlan, or bridge on the deployed host. Once I done that, I would be unable to ssh into the server after reboot. Below is the network layout of my environment. I hided the public IPs, but all else remains the same.
Networks
Native: 192.168.0.0/24, 192.168.24.0/24(ipmi network)
VLAN 100: public network
VLAN 236: 172.29.236.0/22
VLAN 240: 172.29.240.0/22
VLAN 244: 172.29.244.0/22
VLAN 248: 172.29.248.0/22
All switch ports are in trunk mode with the same native vlan id.
MAAS node (all-in-one)
eno1: 192.168.24.200 (ipmi network)
eno1.100: public ip/27
ens2p0: 192.168.0.1 (PXE/base network gateway)
ens2p0.236: 172.29.236.1/22 (reserved)
Deployed servers (one node from each group)
IP is assigned via MAAS as static IP
infra1
eno1: 192.168.0.11
eno2: unconfigured
compute1
ens2p0: 192.168.0.21
eno1: unconfigured
storage1
eno1: 192.168.0.31
eno2: unconfigured
So, doing the ‘ip a s’ on a newly deployed server would look like this (e.g. infra1):
#> ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:25:90:f2:63:1a brd ff:ff:ff:ff:ff:ff
inet 192.168.0.11/24 brd 192.168.0.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fef2:631a/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:25:90:f2:63:1b brd ff:ff:ff:ff:ff:ff
inet6 fe80::225:90ff:fef2:631b/64 scope link
valid_lft forever preferred_lft forever
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:25:90:f2:63:1c brd ff:ff:ff:ff:ff:ff
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:25:90:f2:63:1d brd ff:ff:ff:ff:ff:ff
My intended /etc/network/interfaces for infra1 would look like this:
auto eno1
iface eno1 inet static
address 192.168.0.11
netmask 255.255.255.0
gateway 192.168.0.1
dns-nameservers 8.8.8.8 8.8.4.4
auto eno1.100
iface eno1.100 inet static
address x.x.x.x #a public ip
netmask 255.255.255.224
vlan-raw-device eno1
auto eno1.236
iface eno1.236 inet manual
vlan-raw-device eno1
auto eno1.240
iface eno1.240 inet manual
vlan-raw-device eno1
auto eno2
iface eno2 inet manual
auto eno2.244
iface eno2.244 inet manual
vlan-raw-device eno2
auto br-mgmt
iface br-mgmt inet static
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge_ports eno1.236
address 172.29.236.11
netmask 255.255.252.0
auto br-vxlan
iface br-vxlan inet static
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge_ports eno2.240
address 172.29.240.11
netmask 255.255.252.0
auto br-vlan
iface br-vlan inet manual
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge_ports eno2
auto br-storage
iface br-storage inet manual
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge_ports eno1.244
I have tested the above configurations on both 18.04 LTS and 16.04 LTS. In the case of 18.04, I tried both ifupdown (as above) and netplan (converted to yaml) methods. As soon as I give the eno1.100 a public IP from the same range as used by the MAAS node, I would no longer be able to ssh into the server after reboot. Doing so would return “ssh: connect to host 192.168.0.11 port 22: Connection refused”, although all IPs can be reached by ping. I don’t have a direct console access to servers in the data center at this moment, so I don’t know what happened to them during or after reboot.
As for the output that you asked, here they are:
> maasdb=# \pset pager off
> Pager usage is off.
> maasdb=# select * from maas_support__ip_allocation;
> ip | alloc_type | cidr | hostname | ifid | ifname | iftype | mac_address | power_type
> ----------------+------------+-----------------+----------+------+--------+----------+-------------------+------------
> 192.168.0.1 | STICKY | 192.168.0.0/24 | gw | 2 | enp2s0 | physical | 00:25:90:a2:d1:22 |
> 192.168.0.11 | STICKY | 192.168.0.0/24 | infra1 | 5 | eno1 | physical | 00:25:90:f2:63:1a |
> 192.168.0.12 | STICKY | 192.168.0.0/24 | infra2 | 27 | eno1 | physical | 00:25:90:f3:97:7a |
> 192.168.0.13 | STICKY | 192.168.0.0/24 | infra3 | 33 | eno1 | physical | 00:25:90:f2:58:22 |
> 192.168.0.21 | STICKY | 192.168.0.0/24 | compute1 | 15 | enp2s0 | physical | 00:25:90:a2:d2:c6 |
> 192.168.0.22 | STICKY | 192.168.0.0/24 | compute2 | 19 | enp2s0 | physical | 00:25:90:a1:c6:40 |
> 192.168.0.31 | STICKY | 192.168.0.0/24 | storage1 | 38 | eno1 | physical | 00:25:90:f3:9d:a2 |
> 192.168.0.32 | STICKY | 192.168.0.0/24 | storage2 | 43 | eno1 | physical | 00:25:90:f3:aa:ee |
> 192.168.0.33 | STICKY | 192.168.0.0/24 | storage3 | 47 | eno1 | physical | 00:25:90:f3:7c:a6 |
> 192.168.24.200 | STICKY | 192.168.24.0/24 | gw | 1 | eno1 | physical | 00:25:90:a2:d1:23 |
> 192.168.24.201 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.202 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.203 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.204 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.205 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.206 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.207 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> 192.168.24.208 | STICKY | 192.168.24.0/24 | | | | | | ipmi
> | STICKY | 192.168.0.0/24 | storage2 | 44 | eno2 | physical | 00:25:90:f3:aa:ef |
> | DISCOVERED | 192.168.0.0/24 | infra3 | 34 | eno2 | physical | 00:25:90:f2:58:23 |
> | STICKY | 192.168.0.0/24 | infra3 | 34 | eno2 | physical | 00:25:90:f2:58:23 |
> | DISCOVERED | 192.168.0.0/24 | infra2 | 27 | eno1 | physical | 00:25:90:f3:97:7a |
> | DISCOVERED | 192.168.0.0/24 | infra2 | 28 | eno2 | physical | 00:25:90:f3:97:7b |
> | STICKY | 192.168.0.0/24 | infra2 | 28 | eno2 | physical | 00:25:90:f3:97:7b |
> | DISCOVERED | 192.168.0.0/24 | infra3 | 33 | eno1 | physical | 00:25:90:f2:58:22 |
> | DISCOVERED | 192.168.0.0/24 | infra1 | 5 | eno1 | physical | 00:25:90:f2:63:1a |
> | DISCOVERED | 192.168.0.0/24 | infra1 | 7 | eno2 | physical | 00:25:90:f2:63:1b |
> | STICKY | 192.168.0.0/24 | infra1 | 7 | eno2 | physical | 00:25:90:f2:63:1b |
> | DISCOVERED | 192.168.0.0/24 | compute1 | 16 | eno1 | physical | 00:25:90:a2:d2:c7 |
> | DISCOVERED | 192.168.0.0/24 | compute1 | 15 | enp2s0 | physical | 00:25:90:a2:d2:c6 |
> | STICKY | 192.168.0.0/24 | compute1 | 16 | eno1 | physical | 00:25:90:a2:d2:c7 |
> | DISCOVERED | 192.168.0.0/24 | compute2 | 20 | eno1 | physical | 00:25:90:a1:c6:41 |
> | DISCOVERED | 192.168.0.0/24 | compute2 | 19 | enp2s0 | physical | 00:25:90:a1:c6:40 |
> | STICKY | 192.168.0.0/24 | compute2 | 20 | eno1 | physical | 00:25:90:a1:c6:41 |
> | DISCOVERED | 192.168.0.0/24 | storage3 | 47 | eno1 | physical | 00:25:90:f3:7c:a6 |
> | DISCOVERED | 192.168.0.0/24 | storage3 | 48 | eno2 | physical | 00:25:90:f3:7c:a7 |
> | STICKY | 192.168.0.0/24 | storage3 | 48 | eno2 | physical | 00:25:90:f3:7c:a7 |
> | DISCOVERED | 192.168.0.0/24 | storage1 | 38 | eno1 | physical | 00:25:90:f3:9d:a2 |
> | DISCOVERED | 192.168.0.0/24 | storage1 | 39 | eno2 | physical | 00:25:90:f3:9d:a3 |
> | STICKY | 192.168.0.0/24 | storage1 | 39 | eno2 | physical | 00:25:90:f3:9d:a3 |
> | DISCOVERED | 192.168.0.0/24 | storage2 | 43 | eno1 | physical | 00:25:90:f3:aa:ee |
> | DISCOVERED | 192.168.0.0/24 | storage2 | 44 | eno2 | physical | 00:25:90:f3:aa:ef |This is becoming difficult to diagnose in a forum thread. =)
It’s strange to me that it would work with ping but not ssh. Is it possible that a firewall issue of some kind is preventing communication?
I would take a look at a packet capture to see what might be going wrong. Maybe packets are being rejected due to rp_filter rules or similar issues with asymmetric routing; perhaps a stateful firewall is receiving a packet on an unexpected interface and rejecting the SSH traffic?
@mpontillo it is so weird that the only thing I am ‘forbidden’ to do is to add VLAN 100, which in this case is also used by the all-in-one MAAS server for the public network. If I add VLAN 100 to any interface, vlan interface, or bridge from within the deployed host after deployment, I would be refused to login via ssh after reboot. That said, I can still add VLAN 100 and get everything working on the node after initial deployment - just cannot reboot.
One thing I should point out is that prior to the MAAS installation, I have configured the host as a gateway to forward 192.168.0.0/24 to vlan 100 by using IP masquerading. Not sure if that could be an issue…
@mpontillo Ok, I have found a workaround by doing ‘apt-get purge cloud-init’ .
Previously I tried to remove the datasource from cloud-init by running ‘dpkg-reconfigure cloud-init’, but that didn’t solve the problem.
I think some scripts from cloud-init may have messed up sshd settings after I added a public IP and vlan to the host, but why it only conflicted with public IPs/vlans remains as a mystery to me… 