MAAS 3.1 lxdbr0 routing not working with auto KVM POD deployment

Users
,
1

I deploy baremetal with KVM Pod Setups. Then I snap install LXD then configure LXD using my preseed file.

Create a LXD Container using the lxdbr0 network.

  • can ping google
  • can ping router
  • can’t access local networks

Move LXD to an internal network bridge

  • can ping google
  • can ping router
  • can access everything on local network

I’m not an IP Tables expert but I think there’s an IP Tables rule that might be defaulting all traffic to virbr0 so this is why lxdbr0 containers can’t access anything on the local LAN, only outside into the internet.

I’m not sure how to fix this if the baremetal is installed through MAAS.

I don’t have this problem if I manually install Ubuntu on the baremetal, manually install libvirt and do a LXD Preeseed init.

2

What does your routing table and ip configuration look like on the machine?
What about iptables (iptables -n -L)?
What does the preseed look like?

3

Hi Anton, here’s the info!.

LXD Preeseed

config:
  core.https_address: '[::]:8443'
  core.trust_password: xxxx
  maas.api.key: xxxx
  maas.api.url: http://xxxx:5240/MAAS
networks:
- config:
    ipv4.address: auto
    ipv6.address: none
  description: ""
  name: lxdbr0
  type: ""
storage_pools:
- config:
    source: /share/lxd-storage
  description: ""
  name: default
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      network: lxdbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
cluster: null

On the BareMetal:
ip route

default via 192.168.143.1 dev br-eno1 proto static
10.138.254.0/24 dev lxdbr0 proto kernel scope link src 10.138.254.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
192.168.141.0/24 dev br-eno2.21 proto kernel scope link src 192.168.141.54
192.168.142.0/24 dev br-eno3 proto kernel scope link src 192.168.142.51
192.168.143.0/24 dev br-eno1 proto kernel scope link src 192.168.143.80
192.168.144.0/24 dev br-eno2.24 proto kernel scope link src 192.168.144.33

iptables -n-L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 12 /* generated for LXD network lxdbr0 */
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 11 /* generated for LXD network lxdbr0 */
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 3 /* generated for LXD network lxdbr0 */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdbr0 */
LIBVIRT_INP  all  --  0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
LIBVIRT_FWX  all  --  0.0.0.0/0            0.0.0.0/0
LIBVIRT_FWI  all  --  0.0.0.0/0            0.0.0.0/0
LIBVIRT_FWO  all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 12 /* generated for LXD network lxdbr0 */
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 11 /* generated for LXD network lxdbr0 */
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 3 /* generated for LXD network lxdbr0 */
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdbr0 */
LIBVIRT_OUT  all  --  0.0.0.0/0            0.0.0.0/0

Chain LIBVIRT_FWI (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target     prot opt source               destination
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain LIBVIRT_INP (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:68

iptables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -i lxdbr0 -p icmp -m icmp --icmp-type 12 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p icmp -m icmp --icmp-type 11 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p icmp -m icmp --icmp-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -j LIBVIRT_INP
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -o lxdbr0 -p icmp -m icmp --icmp-type 12 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p icmp -m icmp --icmp-type 11 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p icmp -m icmp --icmp-type 3 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 67 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT

Inside the Container

ip route

default via 10.138.254.1 dev eth0 proto dhcp src 10.138.254.61 metric 100
10.138.254.0/24 dev eth0 proto kernel scope link src 10.138.254.61
10.138.254.1 dev eth0 proto dhcp scope link src 10.138.254.61 metric 100
4

Do you have IPv4 forwarding enabled?

How does your home network return traffic to that range, did you add a route to your router or are you using NAT for traffic from that bridge to external?

5

IPV4 Forwarding is enabled.

$ cat /proc/sys/net/ipv4/ip_forward
1

I have the routes defined on the router too. I know it works because the manual installedUbuntu machines have Libvirt and LXD running inside. LXCs on lxdbr0 can get to the Internal and External network. KVMs on virbr0 can also get to the Internal and External network.

The MAAS deployed machines with LXD in lxdbr0 can’t get to the internal networks, if I move the LXC Container to the bridged network on the internal LAN then it can get into the internal and external networks.

6

when you say you can’t access the local network, do you mean the network:

192.168.143.0/24

can you ping 192.168.143.80 (which looks like the address of your host machine)?

have you tried using tcpdump to see where the traffic is dying? e.g. tcpdump -n -i *interface* icmp and host *a.b.c.d* can help you isolate where the pings are disappearing. Set interface to e.g. lxdbr0, then br-eno1, and also try without specifying the interface, that way you can try to see where the traffic is being dropped.

Also, do you have any nat rules? : iptables -S -t nat

7

I can ping the host, and it also looks like i can ping any IP in the same subnet.

It’s IP’s on different subnets that are unreachable. like 192.168.141.x 192.168.2.x.

Let me try the tcpdump.

iptables -S -t nat

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N LIBVIRT_PRT
-A POSTROUTING -s 10.138.254.0/24 ! -d 10.138.254.0/24 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
-A POSTROUTING -j LIBVIRT_PRT
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
8

@mk4umha, it looks like this one’s still open, am i right?

9

It might be, I haven’t messed with that setup for awhile now. I’ll have to rebuild it to see if the “bug” is still there.

1 Like
10

no pressure. just following up. we don’t do that enough.