Multus CNI with k3s and RKE

01-multus-k3s.md

Using Multus CNI in K3S

By default, K3S will run with flannel as the CNI and use custom directories to store CNI plugin binaries and config files(You can inspect the kubelet args K3S uses via journalctl -u k3s|grep cni-conf-dir). So you need to configure that properly When deploying Multus CNI.

For example given the official Multus manifests in https://github.com/intel/multus-cni/blob/36f2fd64e0965e639a0f1d17ab754f0130951aba/images/multus-daemonset.yml, the following changes are needed:

volumes:
  - name: cni
    hostPath:
      path: /var/lib/rancher/k3s/agent/etc/cni/net.d
  - name: cnibin
    hostPath:
      path: /var/lib/rancher/k3s/data/<replace-with-your-hash>/bin

containers:
  - name: kube-multus
    image: nfvpe/multus:v3.4.1
    command: ["/entrypoint.sh"]
    args:
      - "--multus-conf-file=auto"
      - "--cni-version=0.3.1"
      # Add the following arg
      - "--multus-kubeconfig-file-host=/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig"

Once Multus CNI is deployed properly it works as normal in K3S.

02-multus-rke.md

Using Multus CNI in RKE

1. Provision RKE cluster with automatic CNI deployment disabled

RKE cluster.yaml:

...
network:
    plugin: none
services:
  kubelet:
    extra_args:
      cni-conf-dir: "/etc/cni/net.d"
      cni-bin-dir: "/opt/cni/bin"
...

2. Install Multus

Use the following Helm chart for conveniance:

https://github.com/ibrokethecloud/multus-helm

https://github.com/ibrokethecloud/cni-plugins https://ubuntu.com/kubernetes/docs/cni-sriov

03-external.md

• k8snetworkplumbingwg/multus-cni#539
• https://github.com/kubernetes-sigs/kubespray/blob/master/docs/multus.md
• https://medium.com/@sarpkoksal/multus-installation-on-kubernetes-a0a6ef04972f
• https://github.com/k8snetworkplumbingwg/reference-deployment/blob/a3b4d3066ac0e130deee7d20fa02a72d67e9e683/multus-calico/multus-crio-daemonset.yml
• https://github.com/rancher/kontainer-driver-metadata/blob/497e5d720660bf9c8dc7ab0c7b44ee6daca28534/rke/templates/calico.go
• rancher/rke#873 (comment)

@beefcheeks, thanks for your cni-symlinker service. after struggling mightily with file creation errors, applying your service followed by the multus-cni sample deployment worked great!

@jderose-net you are welcome! I actually now patch it as an initContainer into the multus DaemonSet which eliminates the messiness of ordered dependencies. You can view my patch here.

Just an FYI that for latest K3s if you are using Helm charts you don't need any of this magic:

• Deploy via helm with whereabouts as described here
• don't forget to add specific paths to your NetworkAttachmentDefinition as described in the same url