astoycos

SEC("xdp")
int xdp_nodeport_redirect(struct xdp_md *ctx)
{
  void *data_end = (void *)(long)ctx->data_end;
  void *data = (void *)(long)ctx->data;
  struct bpf_fib_lookup fib_params = {};
  struct ethhdr *eth = data;
  struct tcphdr *tcph;
  u16 h_proto;
  u64 nh_off;

astoycos

SEC("cgroup/skb")
int bpf_sockops_sctp_load(struct __sk_buff *skb)
{
    void *data = (void *)(long)skb->data;
    struct eth_hdr *eth = data;
    void *data_end = (void *)(long)skb->data_end;

    __u32 dstAddr;
    int ret;

astoycos

The Issue

SCTP based connect calls are not being recorded by BPF_PROG_TYPE_CGROUP_SOCK_ADDR BPF programs when attached to the BPF_CGROUP_INET4_CONNECT hook point in the default Cgroup.

As shown below (with strace) all TCP, UDP, and SCTP clients produce connect syscalls when sending data

Syscalls made by TCP Client

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 5

astoycos

type SubRouteKey struct {
	dst string
	gws stringset.Interface
}

func (kp *SyncHandler) listExistingRoutes() (map[SubRouteKey]netlink.Route, error) {
	currentRoutes := map[SubRouteKey]netlink.Route{}

	currentRouteList, err := kp.netLink.RouteList(kp.vxlanDevice.link, syscall.AF_INET)
	if err != nil {

astoycos

1. So Curl Nodeport on another node from pod(server) on node 

[astoycos@fedora kubernetes-debugging-resources]$ oc exec -it server /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@server:/# curl 10.0.176.224:30348
<<TIMEOUT>>>=

[astoycos@fedora kubernetes-debugging-resources]$ oc get pods -o wide
NAME                                            READY   STATUS    RESTARTS   AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
client-agnhost                                  1/1     Running   0          19m     10.128.8.28    ip-10-0-207-198.us-west-2.compute.internal   <none>           <none>

astoycos

~~~~> The first is an error message regarding the pod's annotation which only lasts a few milliseconds (~ 5) 
```
2021-08-10T17:07:52.395332199Z E0810 17:07:52.395292    9598 informer.go:301] error syncing 'solktxesvzwcchf-y-nk-rt-000/solktxesvzwcchf-y-nk-rt-000-cdr-vz-sps-configuration-test-strea': error getting pod details: could not find OVN pod annotation in map[helm.sh/hook:test-success helm.sh/hook-delete-policy:hook-succeeded,before-hook-creation k8s.ovn.org/hybrid-overlay-external-gw:198.18.2.13 k8s.ovn.org/hybrid-overlay-vtep:198.18.0.10 k8s.v1.cni.cncf.io/networks:istio-cni openshift.io/scc:restricted-no-uid0 placeholder:true sidecar.istio.io/inject:true sidecar.istio.io/rewriteAppHTTPProbers:true sidecar.istio.io/status:{"version":"c5f618bb3d10ba61dac06980d4d5d6794e4fc414f780ab398825a2f42f5f9d49","initContainers":["istio-validation"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":["aspen-mesh-pull-secret"]}], req

astoycos

1. 1h3m       Warning  Unhealthy               pod/ovnkube-master-5r8k9         Readiness probe failed: + /usr/bin/ovn-appctl -t /var/run/ovn/ovnsb_db.ctl --timeout=3 cluster/status OVN_Southbound
                                                                             + grep 10.75.214.11
                                                                             + grep -v Address
                                                                             2021-08-02T17:47:35Z|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)



2. Ovnkube master logs on ovnkube-master-5r8k9

2021-08-02T17:47:34.505924011Z I0802 17:47:34.505444       1 master.go:268] Creating / updating node worker-065 hybrid overlay port with mac 0a:58:c0:a8:e4:03

astoycos

docker run -it --rm --security-opt label=disable --cap-add=NET_ADMIN --cap-add=SYS_ADMIN -v /home/astoycos/ovn-kubernetes:/go/src/github.com/ovn-org/ovn-kubernetes -w /go/src/github.com/ovn-org/ovn-kubernetes/go-controller -e COVERALLS= -e GINKGO_FOCUS="" quay.io/giantswarm/golang:1.16.3 sh -c "RACE=1 DOCKER_TEST=1 COVERALLS= PKGS="" hack/test-go.sh focus \"\" "
go test -mod=vendor -test.v -race  github.com/ovn-org/ovn-kubernetes/go-controller/pkg/cni -ginkgo.v  -ginkgo.reportFile ./_artifacts/junit-pkg_cni.xml
W0601 19:15:51.157129   14005 ovs.go:101] Unable to detect OS MAX_ARGS, defaulting to: 262144
=== RUN   TestClearPodBandwidth
=== RUN   TestClearPodBandwidth/0:Test_error_code_path_when_ovsFind_attempts_to_retrieve_interfaces
    bandwidth_test.go:116: PASS:	CombinedOutput()
    bandwidth_test.go:117: PASS:	Command(mock.AnythingOfTypeArgument,mock.AnythingOfTypeArgument,mock.AnythingOfTypeArgument,mock.AnythingOfTypeArgument,mock.AnythingOfTypeArgument,mock.AnythingOfTypeArgument,mock.AnythingOfTypeArg

astoycos

Verifying OVN-K externalTrafficPolicy Feature
1. 2 CLient pods on two different nodes 

[astoycos@nfvsdn-03 demo]$ kubectl get pods -n logging -o wide --show-labels
NAME      READY   STATUS    RESTARTS   AGE    IP           NODE          NOMINATED NODE   READINESS GATES   LABELS
client    1/1     Running   0          112m   10.244.0.4   ovn-worker2   <none>           <none>            app=webserver
client2   1/1     Running   0          106m   10.244.2.6   ovn-worker    <none>           <none>            app=webserver

2. 2 Nodeport services pointing to those pods, one with ETP=Local the other with ETP=Cluster(the default) 

astoycos

1. Create Namespace Logging with ACL activated as follows, with allow and deny set to alert
[astoycos@nfvsdn-03 demo]$ cat log-ns.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: logging
  annotations:
    k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'

## CLient Pods in Both namespace: Default and Logging with a Server Pod in namespace: Logging