Using tc redirect to connect a virtual machine to a container network

tc mirroring.md

Connecting a veth device to tap

• veth device from CNI/CNM plugin: eth0
• tap device that connects to the VM: tap0

Redirecting traffic between the two devices

tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev tap0
tc qdisc add dev tap0 ingress
tc filter add dev tap0 parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev eth0

tc qdisc add dev eth0 ingress

• Add a queuing discipline
• on dev eth0
• attach the ingress qdisc Here the handle defaults to ffff:

tc filter add dev eth0 parent ffff: protocol all u32 match u8 0 0 action mirred egress redirect dev tap0

• Add a filter
• to device dev eth0
• to parent (class) handle to which we are attaching, ffff: i.e. ingress which we created before (there is no need for tc class add in the ingress case as it does not support classful queuing discplines).
• protocol all
• classifier u32
• parameters to the classifier u8 0 0, and the first byte of the packet with 0 and if the result is 0 (which it always will be) (i.e. always true)
• action mirred egress redirect dev eth0, redirect the packet to egress of dev eth0

Go program that implements the above logic:

package main

import (
	"fmt"
	"github.com/vishvananda/netlink"
	"os"
	"strconv"

	"golang.org/x/sys/unix"
)

func main() {
	args := os.Args[1:]

	if len(args) != 2 {
		fmt.Println("Incorrect number of args")
		os.Exit(1)
	}

	index1, _ := strconv.Atoi(args[0])
	index2, _ := strconv.Atoi(args[1])

	fmt.Printf("network index1 : %d\n", index1)
	fmt.Printf("network index2 : %d\n", index2)

	qdisc1 := &netlink.Ingress{
		QdiscAttrs: netlink.QdiscAttrs{
			LinkIndex: index1,
			Parent:    netlink.HANDLE_INGRESS,
		},
	}

	err := netlink.QdiscAdd(qdisc1)
	if err != nil {
		fmt.Printf("Failed to add qdisc for index %d : %s", index1, err)
		os.Exit(1)
	}

	qdisc2 := &netlink.Ingress{
		QdiscAttrs: netlink.QdiscAttrs{
			LinkIndex: index2,
			Parent:    netlink.HANDLE_INGRESS,
		},
	}

	err = netlink.QdiscAdd(qdisc2)
	if err != nil {
		fmt.Printf("Failed to add qdisc for index %d : %s", index2, err)
		os.Exit(1)
	}

	filter1 := &netlink.U32{
		FilterAttrs: netlink.FilterAttrs{
			LinkIndex: index1,
			Parent:    netlink.MakeHandle(0xffff, 0),
			Protocol:  unix.ETH_P_ALL,
		},
		Actions: []netlink.Action{
			&netlink.MirredAction{
				ActionAttrs: netlink.ActionAttrs{
					Action: netlink.TC_ACT_STOLEN,
				},
				MirredAction: netlink.TCA_EGRESS_REDIR,
				Ifindex:      index2,
			},
		},
	}

	if err := netlink.FilterAdd(filter1); err != nil {
		fmt.Printf("Failed to add filter for index %d : %s", index1, err)
		os.Exit(1)
	}

	filter2 := &netlink.U32{
		FilterAttrs: netlink.FilterAttrs{
			LinkIndex: index2,
			Parent:    netlink.MakeHandle(0xffff, 0),
			Protocol:  unix.ETH_P_ALL,
		},
		Actions: []netlink.Action{
			&netlink.MirredAction{
				ActionAttrs: netlink.ActionAttrs{
					Action: netlink.TC_ACT_STOLEN,
				},
				MirredAction: netlink.TCA_EGRESS_REDIR,
				Ifindex:      index1,
			},
		},
	}

	if err := netlink.FilterAdd(filter2); err != nil {
		fmt.Printf("Failed to add filter for index %d : %s", index2, err)
		os.Exit(1)
	}
}

Hi, thank you for that nice trick.
Do you know if this tc based redirect will allow transparent forwarding of layer 2 frames which are filtered on linux bridge by default (like stp bpdu and LACP frames)?

Hi, thank you for that nice trick.
Do you know if this tc based redirect will allow transparent forwarding of layer 2 frames which are filtered on linux bridge by default (like stp bpdu and LACP frames)?

@hellt yes. All traffic should passthro. We use this in Kata containers will all types of CNI interfaces without issues. The performance drop is negleible.

Thanks @mcastelino
that helped a lot in my similar case where I needed to connect qemu tap interfaces to a containers interfaces

thanks for sharing this! I was struggling with lacp over bridge, this helps!

There is a CNI plugin for this https://github.com/awslabs/tc-redirect-tap used with firecracker

There is a CNI plugin for this https://github.com/awslabs/tc-redirect-tap used with firecracker

https://github.com/awslabs/tc-redirect-tap/blob/master/internal/netlink.go#L40 lol

lol indeed.