Kubernetes Multi-Network

Implementation

Kubernetes: https://github.com/LionelJouin/kubernetes/tree/multi-network
- API
  - Based On: https://github.com/kubernetes/enhancements/blob/78f7523bcbcd2caaea16111efb605227a2286ba4/keps/sig-network/3698-multi-network/README.md
- registry, printers and validation
  - From: https://github.com/plwhite/kubernetes/tree/multi-network
  - Set all fields as immutable except enable in PodNetwork Spec
  - Prints Name and specs of PodNetwork and PodNetworkAttachment
- Feature-gate and default PodNetwork
  - New feature gate in pkg/features/kube_features.go
  - New control plane controller (defaultpodnetwork) enabled by default with no provider and kubernetes as name
- Add default PodNetwork to pods and set default PodIPs
  - New admission plugin adding default network (hardcoded network interface eth0 and IsDefaultGW4/IsDefaultGW6 set to true), if not already present, to pods without hostnetwork and validating the default network is present only once.
  - Modified PodIPs validation to accept any number of PodIPs
  - Modified logic to keep the unmanaged PodIPs while setting the pod status and its default PodIPs (pkg/kubelet/kubelet_pods.go)
  - More changes might be required: https://github.com/kubernetes/kubernetes/blob/v1.29.0/pkg/kubelet/kubelet.go#L1759
- Prevent pods to be scheduled if podnetwork is disabled or not existing
  - New prefilter scheduling plugin to check the existence of the PodNetworks attached to a pod and if they are all enabled.
  - Adding Rule in the ClusterRole of the scheduler to be able to read PodNetworks
- Service endpoint using default PodNetwork
  - Modified Endpoint controller to pick only PodIP from default network
  - Modified EndpointSlice controller to pick only PodIP from default network
- PodNetwork status
- Prevent default PodNetwork from deletion
- PodNetworkAttachment handling
- CRI Changes
- Pod printers and kubectl describe
- Probably more (see: https://github.com/kubernetes/enhancements/blob/78f7523bcbcd2caaea16111efb605227a2286ba4/keps/sig-network/3698-multi-network/README.md#attaching-podnetwork-to-a-pod)
Multus-CNI: https://github.com/LionelJouin/multus-cni/tree/multi-network
- Based on: https://github.com/plwhite/multus-cni/tree/podnetwork
- Quick changes to test Multi-Network changes in Kubernetes
  - Get pod networks from its spec instead of annotation (only first ParametersRefs of the PodNetwork is supported)
  - Set PodIPs in the pod status for all PodNetworks managed by Multus (Default Network PodIPs is then not set by Multus)

Build instructions (On Kind)

# Clone the kubernetes fork
git clone -b multi-network [email protected]:LionelJouin/kubernetes.git

# Check and generate files (If any API changes is done)
./hack/update-all.sh

# Build Kind images
kind build node-image . --image kindest/node:multi-network

Kind Config (Follow this for local registry: https://kind.sigs.k8s.io/docs/user/local-registry/):

---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
  kubeProxyMode: ipvs
featureGates:
  "MultiNetwork": true
runtimeConfig:
  "api/alpha": "true"
nodes:
- role: control-plane
  image: kindest/node:multi-network
- role: worker
  image: kindest/node:multi-network

# Clone the Multus fork
git clone -b multi-network [email protected]:LionelJouin/multus-cni.git

# Build Multus
docker build -t localhost:5000/k8snetworkplumbingwg/multus-cni:snapshot-thick-multi-network -f images/Dockerfile.thick .
docker push localhost:5000/k8snetworkplumbingwg/multus-cni:snapshot-thick-multi-network

# Install multus
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/e2e/templates/cni-install.yml.j2 
kubectl apply -f deployments/multus-daemonset-thick.yml

Examples

NetworkAttachmentDefinition

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-nad
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "ranges": [
          [ { "subnet": "169.254.100.0/24" } ],
          [ { "subnet": "100:100::/64" } ]
        ]
      }
    }'

PodNetwork

---
apiVersion: networking.k8s.io/v1alpha1
kind: PodNetwork
metadata:
  name: dataplane
spec:
  enabled: true
  provider: k8s.cni.cncf.io/multus
  parametersRefs:
  - group: "k8s.cni.cncf.io/v1"
    kind: NetworkAttachmentDefinition
    namespace: default
    name: macvlan-nad

Pod with the PodNetwork

---
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  networks:
  - podNetworkName: dataplane
    interfaceName: net1
  containers:
    - name: ubuntu
      image: lioneljouin/example-application:latest # Ubuntu images with more tools
      command:
        - /bin/bash
        - -c
      args:
        - sleep infinity

Deployment with the PodNetwork

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-server
  labels:
    app: ubuntu-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ubuntu-server
  template:
    metadata:
      labels:
        app: ubuntu-server
    spec:
      networks:
      - podNetworkName: dataplane
        interfaceName: net1
      containers:
        - name: ubuntu
          image: lioneljouin/example-application:latest # Ubuntu images with more tools
          command:
            - /bin/bash
            - -c
          args:
            - nc -l -k -p 5000

Service selecting the deployment (Only default PodIPs will be added to the endpoints and endpointslice)

---
apiVersion: v1
kind: Service
metadata:
  name: ubuntu-service
spec:
  selector:
    app: ubuntu-server
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000

Uncertainties

How to handle HostNetwork in networks and PodIPs?
How to fix this conversion: https://github.com/kubernetes/kubernetes/blob/v1.29.0/pkg/apis/core/v1/conversion.go#L259
pkg/kubelet/stats/helper.go#36 could be solved if the default network interface is known
Should the Default network be recognized by its name?
is isDefaultGW4 and isDefaultGW6 set to true on default network?
What should be the Provider and parametersRefs in default PodNetwork?
How to handle multiple parametersRefs?