Multiple vhostuser ports example with testpmd

README.md

testpmd loop with vhost-user-net-plugin example

Multiple vhostuser ports example with testpmd

0. Prerequisites

To follow this example you should have a system with kubernetes available and configured to support native 1 GB hugepages. You should also have multus-cni and vhost-user-net-plugin up and running. See crd-vhostuser-net.yaml for example config to use with multus and fix path to vhost tool. If using OVS, check that you have bridge named br0 in your OVS with ovs-vsctl show and if not, create it with ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev.

1. Build the image to be used

Build image from provided Dockerfile and tag it as ubuntu-dpdk:

docker build . -t ubuntu-dpdk

2. Create pod with multiple vhostuser interfaces

Copy get-prefix.sh script from vhost-user-net-plugin repo to /var/lib/cni/vhostuser/. See pod-multi-vhost.yaml and start the pod:

kubectl create -f pod-multi-vhost.yaml

3. Open terminal to pod and start testpmd

Open terminal to the created pod once it is running:

kubectl exec -it multi-vhost-example bash

Launch testpmd and automatically start forwarding packets after sending first burst:

# Get container ID
export ID=$(/vhu/get-prefix.sh)

# Run testpmd with ports created by vhostplugin
# Note: change coremask to suit your system
testpmd \
    -d librte_pmd_virtio.so.17.11 \
    -m 1024 \
    -c 0xC \
    --file-prefix=testpmd_ \
    --vdev=net_virtio_user0,path=/vhu/${ID}/${ID:0:12}-net1 \
    --vdev=net_virtio_user1,path=/vhu/${ID}/${ID:0:12}-net2 \
    --no-pci \
    -- \
    --no-lsc-interrupt \
    --auto-start \
    --tx-first \
    --stats-period 1 \
    --disable-hw-vlan;

If packets are not going through, you need to configure direct flows to your switch between the used ports. For example, with OVS as the switch, this is done by getting the port numbers with ovs-ofctl dump-ports br0 and configuring flow, for example, from port 1 to port 2 with ovs-ofctl add-flow br0 in_port=1,action=output:2 and vice versa.

crd-vhostuser-net.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: vhostuser-networkobj
spec:
  config: '{
    "cniVersion": "0.3.0",
    "type": "vhostuser",
    "vhost": {
      "vhost_tool": "/opt/cni/bin/ovs-config.py"
    }
  }'

Dockerfile

FROM ubuntu:bionic

RUN apt-get update && apt-get install -y dpdk;

ENTRYPOINT ["bash"]

pod-multi-vhost.yaml

apiVersion: v1
kind: Pod
metadata:
  name: multi-vhost-example
  annotations:
    k8s.v1.cni.cncf.io/networks: vhostuser-networkobj, vhostuser-networkobj
spec:
  containers:
  - name: multi-vhost-example
    image: ubuntu-dpdk
    imagePullPolicy: Never
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /vhu/
      name: socket
    - mountPath: /mnt/huge
      name: hugepage
    resources:
      requests:
        memory: 1Gi
      limits:
        hugepages-1Gi: 1Gi
    command: ["sleep", "infinity"]
  volumes:
  - name: socket
    hostPath:
      path: /var/lib/cni/vhostuser/
  - name: hugepage
    emptyDir:
      medium: HugePages

@kangasta : I really like to include this part in the read me file. If you send me a PR, I will merge it . Please let me know you thoughts on it.

@rkamudhan: Updated now the example to use latest multus and NetworkAttachmentDefinition standard. I will prepare PR.

PR now submitted