Can not add a Bridge and configure it's IP at the same time


#1

This is fine but, a bit annoying.

While creating a Bridge, it’s “possible” to configure the IP, like static, for example. However, after that, the IP disappear. Then, I have to re-add the IP!

So, instead of one step (add bridge + its IP), there is a need to do it in two steps (add a bridge, then, after it’s there, add the IP).

This looks like a bug, is it already reported?


#2

Yes, this is a known issue. We looked at fixing this during the 2.5 cycle, but encountered some blocking issues; issues; see the merge proposal here for additional discussion.

See also: the bug I filed about this.


#3

@tmartins I am doing the same as you are - using MAAS to prepare metals for OSA.

An issue I am facing right now is that I cannot add static IPs to VLANs on the interface that previously used by MAAS (2.4.2) during metal deployment.

E.g. if eno1 was assigned a static IP by MAAS for provisioning, then I cannot add IPs to VLAN like eno1.xxx. Otherwise, I will no longer be able to ssh into the server after reboot (showing ‘Connection refused’ even the IP is up).


#4

@codylab, this sounds like a separate issue, but I have a couple of questions about your situation.

  1. Do you delete the IP assignment from the non-VLAN interface before assigning it to the VLAN interface?

  2. If the answer to (1) is “yes” (or not applicable), can you take a look at the output of the following command (when run on a MAAS region server) and let me know where the IP address in question appears in the output?

    sudo maas-region dbshell
    \pset pager off
    select * from maas_support__ip_allocation;
    \q


#5

@mpontillo Thanks for checking in. I have narrowed down the issue since the last post. Everything works fine until I add IP from the public network to any interface, vlan, or bridge on the deployed host. Once I done that, I would be unable to ssh into the server after reboot. Below is the network layout of my environment. I hided the public IPs, but all else remains the same.

Networks
Native: 192.168.0.0/24, 192.168.24.0/24(ipmi network)
VLAN 100: public network
VLAN 236: 172.29.236.0/22
VLAN 240: 172.29.240.0/22
VLAN 244: 172.29.244.0/22
VLAN 248: 172.29.248.0/22

All switch ports are in trunk mode with the same native vlan id.

MAAS node (all-in-one)
eno1: 192.168.24.200 (ipmi network)
eno1.100: public ip/27
ens2p0: 192.168.0.1 (PXE/base network gateway)
ens2p0.236: 172.29.236.1/22 (reserved)

Deployed servers (one node from each group)
IP is assigned via MAAS as static IP

infra1
eno1: 192.168.0.11
eno2: unconfigured

compute1
ens2p0: 192.168.0.21
eno1: unconfigured

storage1
eno1: 192.168.0.31
eno2: unconfigured

So, doing the ‘ip a s’ on a newly deployed server would look like this (e.g. infra1):

#> ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:25:90:f2:63:1a brd ff:ff:ff:ff:ff:ff
inet 192.168.0.11/24 brd 192.168.0.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fef2:631a/64 scope link
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:25:90:f2:63:1b brd ff:ff:ff:ff:ff:ff
inet6 fe80::225:90ff:fef2:631b/64 scope link
valid_lft forever preferred_lft forever
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:25:90:f2:63:1c brd ff:ff:ff:ff:ff:ff
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:25:90:f2:63:1d brd ff:ff:ff:ff:ff:ff

My intended /etc/network/interfaces for infra1 would look like this:

auto eno1
iface eno1 inet static
  address 192.168.0.11
  netmask 255.255.255.0
  gateway 192.168.0.1
  dns-nameservers 8.8.8.8 8.8.4.4

auto eno1.100
iface eno1.100 inet static
  address x.x.x.x  #a public ip
  netmask 255.255.255.224
  vlan-raw-device eno1

auto eno1.236
iface eno1.236 inet manual
  vlan-raw-device eno1

auto eno1.240
iface eno1.240 inet manual
  vlan-raw-device eno1

auto eno2
iface eno2 inet manual

auto eno2.244
iface eno2.244 inet manual
  vlan-raw-device eno2

auto br-mgmt
iface br-mgmt inet static
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    bridge_ports eno1.236
    address 172.29.236.11
    netmask 255.255.252.0

auto br-vxlan
iface br-vxlan inet static
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    bridge_ports eno2.240
    address 172.29.240.11
    netmask 255.255.252.0

auto br-vlan
iface br-vlan inet manual
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    bridge_ports eno2

auto br-storage
iface br-storage inet manual
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    bridge_ports eno1.244

I have tested the above configurations on both 18.04 LTS and 16.04 LTS. In the case of 18.04, I tried both ifupdown (as above) and netplan (converted to yaml) methods. As soon as I give the eno1.100 a public IP from the same range as used by the MAAS node, I would no longer be able to ssh into the server after reboot. Doing so would return “ssh: connect to host 192.168.0.11 port 22: Connection refused”, although all IPs can be reached by ping. I don’t have a direct console access to servers in the data center at this moment, so I don’t know what happened to them during or after reboot.

As for the output that you asked, here they are:

> maasdb=# \pset pager off
> Pager usage is off.
> maasdb=# select * from maas_support__ip_allocation;
>        ip       | alloc_type |      cidr       | hostname | ifid | ifname |  iftype  |    mac_address    | power_type 
> ----------------+------------+-----------------+----------+------+--------+----------+-------------------+------------
>  192.168.0.1    | STICKY     | 192.168.0.0/24  | gw       |    2 | enp2s0 | physical | 00:25:90:a2:d1:22 | 
>  192.168.0.11   | STICKY     | 192.168.0.0/24  | infra1   |    5 | eno1   | physical | 00:25:90:f2:63:1a | 
>  192.168.0.12   | STICKY     | 192.168.0.0/24  | infra2   |   27 | eno1   | physical | 00:25:90:f3:97:7a | 
>  192.168.0.13   | STICKY     | 192.168.0.0/24  | infra3   |   33 | eno1   | physical | 00:25:90:f2:58:22 | 
>  192.168.0.21   | STICKY     | 192.168.0.0/24  | compute1 |   15 | enp2s0 | physical | 00:25:90:a2:d2:c6 | 
>  192.168.0.22   | STICKY     | 192.168.0.0/24  | compute2 |   19 | enp2s0 | physical | 00:25:90:a1:c6:40 | 
>  192.168.0.31   | STICKY     | 192.168.0.0/24  | storage1 |   38 | eno1   | physical | 00:25:90:f3:9d:a2 | 
>  192.168.0.32   | STICKY     | 192.168.0.0/24  | storage2 |   43 | eno1   | physical | 00:25:90:f3:aa:ee | 
>  192.168.0.33   | STICKY     | 192.168.0.0/24  | storage3 |   47 | eno1   | physical | 00:25:90:f3:7c:a6 | 
>  192.168.24.200 | STICKY     | 192.168.24.0/24 | gw       |    1 | eno1   | physical | 00:25:90:a2:d1:23 | 
>  192.168.24.201 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.202 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.203 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.204 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.205 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.206 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.207 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>  192.168.24.208 | STICKY     | 192.168.24.0/24 |          |      |        |          |                   | ipmi
>                 | STICKY     | 192.168.0.0/24  | storage2 |   44 | eno2   | physical | 00:25:90:f3:aa:ef | 
>                 | DISCOVERED | 192.168.0.0/24  | infra3   |   34 | eno2   | physical | 00:25:90:f2:58:23 | 
>                 | STICKY     | 192.168.0.0/24  | infra3   |   34 | eno2   | physical | 00:25:90:f2:58:23 | 
>                 | DISCOVERED | 192.168.0.0/24  | infra2   |   27 | eno1   | physical | 00:25:90:f3:97:7a | 
>                 | DISCOVERED | 192.168.0.0/24  | infra2   |   28 | eno2   | physical | 00:25:90:f3:97:7b | 
>                 | STICKY     | 192.168.0.0/24  | infra2   |   28 | eno2   | physical | 00:25:90:f3:97:7b | 
>                 | DISCOVERED | 192.168.0.0/24  | infra3   |   33 | eno1   | physical | 00:25:90:f2:58:22 | 
>                 | DISCOVERED | 192.168.0.0/24  | infra1   |    5 | eno1   | physical | 00:25:90:f2:63:1a | 
>                 | DISCOVERED | 192.168.0.0/24  | infra1   |    7 | eno2   | physical | 00:25:90:f2:63:1b | 
>                 | STICKY     | 192.168.0.0/24  | infra1   |    7 | eno2   | physical | 00:25:90:f2:63:1b | 
>                 | DISCOVERED | 192.168.0.0/24  | compute1 |   16 | eno1   | physical | 00:25:90:a2:d2:c7 | 
>                 | DISCOVERED | 192.168.0.0/24  | compute1 |   15 | enp2s0 | physical | 00:25:90:a2:d2:c6 | 
>                 | STICKY     | 192.168.0.0/24  | compute1 |   16 | eno1   | physical | 00:25:90:a2:d2:c7 | 
>                 | DISCOVERED | 192.168.0.0/24  | compute2 |   20 | eno1   | physical | 00:25:90:a1:c6:41 | 
>                 | DISCOVERED | 192.168.0.0/24  | compute2 |   19 | enp2s0 | physical | 00:25:90:a1:c6:40 | 
>                 | STICKY     | 192.168.0.0/24  | compute2 |   20 | eno1   | physical | 00:25:90:a1:c6:41 | 
>                 | DISCOVERED | 192.168.0.0/24  | storage3 |   47 | eno1   | physical | 00:25:90:f3:7c:a6 | 
>                 | DISCOVERED | 192.168.0.0/24  | storage3 |   48 | eno2   | physical | 00:25:90:f3:7c:a7 | 
>                 | STICKY     | 192.168.0.0/24  | storage3 |   48 | eno2   | physical | 00:25:90:f3:7c:a7 | 
>                 | DISCOVERED | 192.168.0.0/24  | storage1 |   38 | eno1   | physical | 00:25:90:f3:9d:a2 | 
>                 | DISCOVERED | 192.168.0.0/24  | storage1 |   39 | eno2   | physical | 00:25:90:f3:9d:a3 | 
>                 | STICKY     | 192.168.0.0/24  | storage1 |   39 | eno2   | physical | 00:25:90:f3:9d:a3 | 
>                 | DISCOVERED | 192.168.0.0/24  | storage2 |   43 | eno1   | physical | 00:25:90:f3:aa:ee | 
>                 | DISCOVERED | 192.168.0.0/24  | storage2 |   44 | eno2   | physical | 00:25:90:f3:aa:ef |

#6

This is becoming difficult to diagnose in a forum thread. =)

It’s strange to me that it would work with ping but not ssh. Is it possible that a firewall issue of some kind is preventing communication?

I would take a look at a packet capture to see what might be going wrong. Maybe packets are being rejected due to rp_filter rules or similar issues with asymmetric routing; perhaps a stateful firewall is receiving a packet on an unexpected interface and rejecting the SSH traffic?


#7

@mpontillo it is so weird that the only thing I am ‘forbidden’ to do is to add VLAN 100, which in this case is also used by the all-in-one MAAS server for the public network. If I add VLAN 100 to any interface, vlan interface, or bridge from within the deployed host after deployment, I would be refused to login via ssh after reboot. That said, I can still add VLAN 100 and get everything working on the node after initial deployment - just cannot reboot.

One thing I should point out is that prior to the MAAS installation, I have configured the host as a gateway to forward 192.168.0.0/24 to vlan 100 by using IP masquerading. Not sure if that could be an issue…


#8

@mpontillo Ok, I have found a workaround by doing ‘apt-get purge cloud-init’ .

Previously I tried to remove the datasource from cloud-init by running ‘dpkg-reconfigure cloud-init’, but that didn’t solve the problem.

I think some scripts from cloud-init may have messed up sshd settings after I added a public IP and vlan to the host, but why it only conflicted with public IPs/vlans remains as a mystery to me… :roll_eyes: