Using VLANs with bridging with KVM always seems to confuse me.
I dont do it enough and tend to overthink it.
I always search to find out how to do it using the nmcli command.
But finding this was not as easy as it was in the past now that vlan filtering is available inside the bridge.
That's another monster to figure out when I have time.
I always write this down somewhere and always seem to misplace it. I decided a Gist is the perfect place to not lose it. So here goes.
Note: I use three VLANs here. One of the VLANs ihas an ID of 3800. In this document I mislabeled bridges and interfaces concerning VLAN 3800 as STORAGE. This VLAN is not used for storage. But the other two VLANs are used for storage.
The environment I am currently using is a physical server with multiple physical network interfaces. All but two interfaces are disabled/disconnected.
The first interface eno1np0 is connected to the vlan 3800 which is the default vlan used to access the servers in the lab. I have two other vlans, 3810 and 3820, that the VMs will need access to using the eno2np1 interface.
| VLAN ID | CIDR |
|---|---|
| 3800 | 10.100.0.0/16 |
| 3810 | 172.16.0.0/16 |
| 3820 | 192.168.0.0/16 |
Currently the interface configuration look as follows:
# ip address
3: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:02:02:11:22:33 brd ff:ff:ff:ff:ff:ff
altname enp23s0f0np0
altname ens1f0np0
inet 10.100.76.40/24 brd 10.100.76.255 scope global dynamic noprefixroute eno1np0
valid_lft 543sec preferred_lft 543sec
4: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
altname enp23s0f1np1
altname ens1f1np1
# nmcli connection
NAME UUID TYPE DEVICE
eno2np1 c19a4330-91e7-3349-8b1b-2e588f2f8642 ethernet eno2np1
eno1np0 76bb1543-0e4a-3a7c-a4f5-ee82e0df2e39 ethernet eno1np0
lo c6759d6f-7b36-407e-88bc-2555fad7553b loopback lo
virbr0 8e005b60-7b8a-4c6d-8966-58060f4bc723 bridge virbr0
# nmcli device show
DEVICE TYPE STATE CONNECTION
eno1np0 ethernet connected eno1np0
lo loopback connected (externally) lo
virbr0 bridge connected (externally) virbr0
eno2np1 ethernet connecting (getting IP configuration) eno2np1
I would like to add all the vlans to the eno2np1 interface. This includes VLAN 3800.
The eno1np0 interface will remain unchanges and will be used to access the KVM server. This is not really needed because we could use the VLAN 3800 that is on eno2np1.
Note: The eno1np0 interface is connected to a switch port that is in access mode and is tied to only VLAN 3800 on the switch.
Note: The eno2np1 interface needs to be attached to a switch port that is configured for trunking mode.
To put this simply, all we are going to do is create a bridge for each VLAN and then create a VLAN on the eno2np1 port that uses the bridge as its master. It is really as simple as that. Don't overthink this as I usually do.
Let's configure VLAN 3810 first.
First create the bridge for it.
nmcli connection add type bridge con-name storage-3810 ifname storage-3810-if ip4 172.16.76.40/16
A few things to note about the above command:
- con-name is the name for the connection. This shows up in the
nmcli connectionscommand. - ifname is the name of the interface to be created. It can be the same as the con-name, but I find this confusing when troubleshooting. This shows up in the
nmcli devicescommand. - ip4 is a static IP address to setup for this interface. Excluding this will have the interface get an address via DHCP. I want an address on this interface because I also want the KVM host to access systems on the VLAN.
Now we need to add VLAN 3810 and tell it to attach to bridge storage-3810
nmcli con add type vlan id 3810 con-name eno2np1.3810 ifname eno2np1.3810-if dev eno2np1 master storage-3810
A few notes about the above VLAN command:
- id is the VLAN number to add.
- con-name is the name for the VLAN. Using the interface name and VLAN number makes it easier to identify.
- ifname is the name for the interface to be created. It can also be the same as con-name, but I like to change it for troubleshooting purposes. This shows up in the
nmcli devicescommand. - dev is the device/interface to create the VLAN on. In our case its eno2np1.
- master specifies the bridge to tie this VLAN to.
Finally, bring up the VLAN interface and the bridge interface.
nmcli c up eno2np1.3810
nmcli c up storage-3810
That is all there is to doing this. Lets add the bridge and VLAN for VLAN ID 3820. It will be the same as we did for VLAN ID 3810, except the names will change to protect the innocent. - That is a funny. You should laugh.
Configuration for VLAN ID 3820:
nmcli connection add type bridge con-name storage-3820 ifname storage-3820-if ip4 192.168.76.40/16
nmcli con add type vlan id 3820 con-name eno2np1.3820 ifname eno2np1.3820-if dev eno2np1 master storage-3820
nmcli c up eno2np1.3820
nmcli c up storage-3820
Ok, lets configure VLAN ID 3800 on the eno2np1 interface. But we do not want to add an IP address to the bridge because we do not need the KVM host to talk out the eno2np1 interface to the network on VLAN 3800. We want the KVM host to use the eno1np0 interface to VLAN 3800. Again, this was a design decision and we cold easily configure an IP on this bridge for VLAN 3800 and shutdown the eno1np0 interface.
We only want one interface for VLAN 3800 to be used by the KVM host. If two interfaces have IPs on VLAN 3800 then we could cause a routing loop. It's not a good thing.
Configuration for VLAN ID 3800:
nmcli connection add type bridge con-name storage-3800 ifname storage-3800-if
nmcli c mod storage-3800 ipv4.method disabled
nmcli con add type vlan id 3800 con-name eno2np1.3800 ifname eno2np1.3800-if dev eno2np1 master storage-3800
nmcli c up eno2np1.3800
nmcli c up storage-3800
Did you notice that there was an extra command in there? It is to disable DHCP on the bridge. Remember, we do not want the bridge to get an IP address.
After the configuration is done, the interface configuration should look as follows.
# ip address
3: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:02:02:11:22:33 brd ff:ff:ff:ff:ff:ff
altname enp23s0f0np0
altname ens1f0np0
inet 10.100.76.40/24 brd 10.100.76.255 scope global dynamic noprefixroute eno1np0
valid_lft 543sec preferred_lft 543sec
4: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
altname enp23s0f1np1
altname ens1f1np1
21: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:ec:39:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
58: storage-3810-if: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
inet 172.16.76.40/16 brd 172.16.255.255 scope global noprefixroute storage-1234-if
valid_lft forever preferred_lft forever
60: eno2np1.3810-if@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master storage-1234-if state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
61: storage-3820-if: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
inet 192.168.76.40/16 brd 192.168.255.255 scope global noprefixroute storage-3950-if
valid_lft forever preferred_lft forever
63: eno2np1.3820-if@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master storage-3950-if state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
66: storage-3800-if: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
68: eno2np1.3800-if@eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master storage-343-if state UP group default qlen 1000
link/ether 02:02:02:11:22:44 brd ff:ff:ff:ff:ff:ff
# nmcli c
NAME UUID TYPE DEVICE
eno2np1 c19a4330-91e7-3349-8b1b-2e588f2f8642 ethernet eno2np1
eno1np0 76bb1543-0e4a-3a7c-a4f5-ee82e0df2e39 ethernet eno1np0
lo c6759d6f-7b36-407e-88bc-2555fad7553b loopback lo
storage-1234 4cb71edb-bfb6-483f-bbce-6351008619d0 bridge storage-1234-if
storage-343 b546b727-5b2d-4bfb-ad88-fff36cf12258 bridge storage-343-if
storage-3950 7c7d1b65-137e-4554-bf88-bbd3e0eb10d5 bridge storage-3950-if
virbr0 8e005b60-7b8a-4c6d-8966-58060f4bc723 bridge virbr0
eno2np1.1234 65ede8a5-e58e-4d31-a9da-3df65e80323e vlan eno2np1.1234-if
eno2np1.343 97a10674-1513-498a-b4de-b7e48af25e1a vlan eno2np1.343-if
eno2np1.3950 4ef62a8d-3df2-4ed7-9672-673626f284dd vlan eno2np1.3950-if
# nmcli device
DEVICE TYPE STATE CONNECTION
eno1np0 ethernet connected eno1np0
storage-1234-if bridge connected storage-1234
storage-343-if bridge connected storage-343
storage-3950-if bridge connected storage-3950
lo loopback connected (externally) lo
virbr0 bridge connected (externally) virbr0
eno2np1.1234-if vlan connected eno2np1.1234
eno2np1.343-if vlan connected eno2np1.343
eno2np1.3950-if vlan connected eno2np1.3950
eno2np1 ethernet connecting (getting IP configuration) eno2np1
From the KVM host command line, we can ping and see if and how the traffic flows.
I am using the -R option to the ping command.
This will show interface hops.
See the manual page on ping for the caveats for using it.
Especially if its not working for you.
For VLAN 3800 from the KVM host:
# ping -nRc 1 10.100.73.101
PING 10.100.73.101 (10.100.73.101) 56(124) bytes of data.
64 bytes from 10.100.73.101: icmp_seq=1 ttl=63 time=1.80 ms
RR: 10.100.76.40
10.100.73.252
10.100.73.101
10.100.73.101
10.100.76.253
10.100.76.40
--- 10.100.73.101 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.797/1.797/1.797/0.000 ms
We can see that the traffic is going out the eno1np0 interface because the first hop is 10.100.76.40.
For VLAN 3810 from the KVM host:
# ping -nR -c 1 172.16.130.10
PING 172.16.130.10 (172.16.130.10) 56(124) bytes of data.
64 bytes from 172.16.130.10: icmp_seq=1 ttl=64 time=0.138 ms
RR: 172.16.76.40
172.16.130.10
172.16.76.40
--- 172.16.130.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.138/0.138/0.138/0.000 ms
For VLAN 3820 from the KVM host:
# ping -nR -c 1 192.168.130.10
PING 192.168.130.10 (192.168.130.10) 56(124) bytes of data.
64 bytes from 192.168.130.10: icmp_seq=1 ttl=64 time=0.381 ms
RR: 192.168.76.40
192.168.130.10
192.168.76.40
--- 192.168.130.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.381/0.381/0.381/0.000 ms
To attach a VM to the VLAN, create a bridged interface in the VM that points to the bridges interface, storage-3800-if and not its connection.
<interface type="bridge">
<source bridge="storage-3950-if"/>
<model type="virtio"/>
</interface>
You can then use the ping -R command inside the VM to verify connectivity.
Setting routes and gateways (and other items like MTU) can be done on the bridge using the nmcli modify <bridge connection>.
Search for this on-line or in the documentation on how to do this ans what to do.
Its pretty simple.