Sniff CNI commands

README.md

CNI SNIFFER

The Container Networking Interface, or CNI, is a generic plugin-based networking solution for configuring the network on containers.

The CNI specification defines:

1. A format for administrators to define network configuration.
2. A protocol for container runtimes to make requests to network plugins.
3. A procedure for executing plugins based on a supplied configuration.
4. A procedure for plugins to delegate functionality to other plugins.
5. Data types for plugins to return their results to the runtime.

The CNI protocol is based on execution of binaries invoked by the container runtime. CNI defines the protocol between the plugin binary and the runtime. The runtime passes parameters to the plugin via environment variables and configuration. It supplies configuration via stdin. The plugin returns a result on stdout on success, or an error on stderr if the operation fails. Configuration and results are encoded in JSON.

See https://github.com/containernetworking/cni/blob/main/SPEC.md for the complete details.

Sniffer

Since is hard to debug the CNI problems, an useful trick is to replace the CNI binary by a wrapper script that logs the stdin, stdout, stderr and environments variables and pipes them to the CNI binary.

IMPORTANT

This trick can be used with malicious intentions too, it is important you define the corresponding permissions on the CNI binary folders to avoid this vector attack.

Demo

We are going to use a Kind cluster and log into one of the nodes:

$ kind create cluster
$ docker exec -it kind-control-plane

The CNI configuration file defines the plugins to be used:

$ more /etc/cni/net.d/10-kindnet.conflist

{
        "cniVersion": "0.3.1",
        "name": "kindnet",
        "plugins": [
        {
                "type": "ptp",
                "ipMasq": false,
                "ipam": {
                        "type": "host-local",
                        "dataDir": "/run/cni-ipam-state",
                        "routes": [


                                { "dst": "0.0.0.0/0" }
                        ],
                        "ranges": [


                                [ { "subnet": "10.244.0.0/24" } ]
                        ]
                }
                ,
                "mtu": 1500

        },
        {
                "type": "portmap",
                "capabilities": {
                        "portMappings": true
                }
        }
        ]
}

We can see in this configuration that the following plugins: ptp, host-local and portmap are being used.

We can replace one of the binaries by the cni_sniffer.sh (configure the scripts variables accordenly)

$ ls /opt/cni/bin/
host-local  loopback  portmap  ptp
$ mv /opt/cni/bin/ptp /opt/cni/bin/ptp.orig
$ cat cni_sniffer.sh > /opt/cni/bin/ptp
$ chmod +x /opt/cni/bin/ptp

Create a pod:

kubectl run test --image busybox -- sleep 100

And observe the CNI commands tailing the log file:

$ tail -f /tmp/cni.log
ENV:
 HTTPS_PROXY=
LD_LIBRARY_PATH=/opt/containerd/lib:
NO_PROXY=
CNI_ARGS=K8S_POD_UID=a549fbe5-bc12-4a3f-a837-addd0243dc29;IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=test;K8S_POD_INFRA_CONTAINER_ID=fe13dca1c089005b13daf40c025c0872d97f71469fba695b8de163426ccf5a23
CNI_PATH=/opt/cni/bin
SYSTEMD_EXEC_PID=105
JOURNAL_STREAM=8:1267825
CNI_CONTAINERID=fe13dca1c089005b13daf40c025c0872d97f71469fba695b8de163426ccf5a23
PATH=/opt/containerd/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/nri/bin
INVOCATION_ID=36d4ea5c063e48fbbf131b7e4d9edceb
CNI_NETNS=/var/run/netns/cni-eca98d98-a4de-b5e1-3609-005fec750fa0
CNI_IFNAME=eth0
LANG=C.UTF-8
CNI_COMMAND=ADD
PWD=/
HTTP_PROXY=
STDIN:
 {"cniVersion":"0.3.1","ipMasq":false,"ipam":{"dataDir":"/run/cni-ipam-state","ranges":[[{"subnet":"10.244.0.0/24"}]],"routes":[{"dst":"0.0.0.0/0"}],"type":"host-local"},"mtu":1500,"name":"kindnet","type":"ptp"}
STDOUT:
 {
    "cniVersion": "0.3.1",
    "interfaces": [
        {
            "name": "veth12c27226",
            "mac": "fa:24:e0:52:1f:14"
        },
        {
            "name": "eth0",
            "mac": "ae:75:ea:6c:8b:8d",
            "sandbox": "/var/run/netns/cni-eca98d98-a4de-b5e1-3609-005fec750fa0"
        }
    ],
    "ips": [
        {
            "version": "4",
            "interface": 1,
            "address": "10.244.0.43/24",
            "gateway": "10.244.0.1"
        }
    ],
    "routes": [
        {
            "dst": "0.0.0.0/0"
        }
    ],
    "dns": {}
}
STDERR:

BONUS

Now that you are in the middle you can use jq to filter the input and output and use it to inject failures, ... your imagination is the limit at this point.

cni_sniffer.sh

#!/bin/sh

CNI_BIN=/opt/cni/bin/ptp.bak
LOG_FILE=/tmp/cni.log

# Capture input
STD_IN=$(cat)
ENV=$(env)

echo "ENV:\n $ENV" >> $LOG_FILE
echo "STDIN:\n $STD_IN" >> $LOG_FILE

# xref: http://mywiki.wooledge.org/BashFAQ/002
# What you cannot do is capture stdout in one variable, and stderr in another, using only FD redirections.
# You must use a temporary file (or a named pipe) to achieve that one.

result=$(
    { stdout=$(echo $STD_IN | $CNI_BIN); returncode=$?; } 2>&1
    printf "this is the separator"
    printf "%s\n" "$stdout"
    exit "$returncode"
)

STATUS=$?
STD_OUT=${result#*this is the separator}
STD_ERR=${result%this is the separator*}

echo "STDOUT:\n $STD_OUT" >> $LOG_FILE
echo "STDERR:\n $STD_ERR" >> $LOG_FILE

echo $STD_OUT
echo $STD_ERR >&2
exit $STATUS