Skip to content

Instantly share code, notes, and snippets.

@jamesharr

jamesharr/000-README.md

Last active December 23, 2021 00:03
Show Gist options
  • Save jamesharr/5d91396f62d1220e5cf898cffce0e32c to your computer and use it in GitHub Desktop.
Save jamesharr/5d91396f62d1220e5cf898cffce0e32c to your computer and use it in GitHub Desktop.
Download ZIP
Cilium DNS Test Problems
Raw
000-README.md

I'm having a problem getting FQDNs working, and I figured I'd post here to see if anyone could help.

The logs I'm seeing from hubble observe seem to fly directly in the face of the policy that's written.

The Kubernetes manifest is attached as dns-return-test.yml. The cluster is K3s (latest) running fairly stock. See below for details.

Things that seemed odd to me:

  • I had to explicitly allow return traffic from kube-dns to my pods, if that was not in the manifest, DNS would not work and hubble observe would show a weird ERROR. The specific area of the manifest is noted.
  • Disabling DNS inspection (noted in manifest) would cause DNS to work fine without the aforementioned rule. FQDNs wouldn't work, but DNS would.
% k apply -f dns-test.yml              
pod/dns-client created
networkpolicy.networking.k8s.io/default-ingress-deny created
ciliumnetworkpolicy.cilium.io/example-egress-policy created

% k exec -it dns-client -- curl -4 example.com
curl: (7) Failed to connect to example.com port 80: Connection timed out
command terminated with exit code 7
Raw
dns-test.yml
---
apiVersion: v1
kind: Pod
metadata:
name: dns-client
labels:
name: dns-client
spec:
containers:
- name: dns-client
# Image is just an ubuntu image that has a pile of network tools like
# dig, nmap, fping, tcpdump, and some other tools in it in it.
image: jamesharr/ubuntu-netshoot:latest
command:
- bash
args:
- -lc
- "sleep infinity"
resources:
limits:
memory: "128Mi"
cpu: "500m"
terminationGracePeriodSeconds: 1
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-ingress-deny
spec:
podSelector: {}
policyTypes:
- Ingress
### The following seems to be required for DNS to work (at all)
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "example-egress-policy"
spec:
endpointSelector:
matchLabels:
name: dns-client
egress:
- toFQDNs:
- matchName: "example.com"
toPorts:
- ports:
- port: "80"
protocol: TCP
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
"k8s:k8s-app": kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
Raw
z01-hubble-observe.txt
% hubble observe -n cilium-test1 -f
Dec 22 23:47:16.765: cilium-test1/dns-client:45049 -> kube-system/coredns-85cb69466-2ff7k:53 L3-L4 REDIRECTED (UDP)
Dec 22 23:47:16.765: cilium-test1/dns-client:45049 -> kube-system/coredns-85cb69466-2ff7k:53 to-proxy FORWARDED (UDP)
Dec 22 23:47:16.765: cilium-test1/dns-client:45049 -> kube-system/coredns-85cb69466-2ff7k:53 dns-request FORWARDED (DNS Query example.com.cilium-test1.svc.cluster.local. A)
Dec 22 23:47:16.765: cilium-test1/dns-client:45049 <- kube-system/coredns-85cb69466-2ff7k:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy example.com.cilium-test1.svc.cluster.local. A))
Dec 22 23:47:16.765: cilium-test1/dns-client:45049 <- kube-system/coredns-85cb69466-2ff7k:53 to-endpoint FORWARDED (UDP)
Dec 22 23:47:16.765: cilium-test1/dns-client:35656 -> kube-system/coredns-85cb69466-2ff7k:53 L3-L4 REDIRECTED (UDP)
Dec 22 23:47:16.765: cilium-test1/dns-client:35656 -> kube-system/coredns-85cb69466-2ff7k:53 to-proxy FORWARDED (UDP)
Dec 22 23:47:16.765: cilium-test1/dns-client:35656 -> kube-system/coredns-85cb69466-2ff7k:53 dns-request FORWARDED (DNS Query example.com.svc.cluster.local. A)
Dec 22 23:47:16.766: cilium-test1/dns-client:35656 <- kube-system/coredns-85cb69466-2ff7k:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy example.com.svc.cluster.local. A))
Dec 22 23:47:16.766: cilium-test1/dns-client:35656 <- kube-system/coredns-85cb69466-2ff7k:53 to-endpoint FORWARDED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:33796 -> kube-system/coredns-85cb69466-2ff7k:53 L3-L4 REDIRECTED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:33796 -> kube-system/coredns-85cb69466-2ff7k:53 to-proxy FORWARDED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:33796 -> kube-system/coredns-85cb69466-2ff7k:53 dns-request FORWARDED (DNS Query example.com.cluster.local. A)
Dec 22 23:47:16.766: cilium-test1/dns-client:33796 <- kube-system/coredns-85cb69466-2ff7k:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy example.com.cluster.local. A))
Dec 22 23:47:16.766: cilium-test1/dns-client:33796 <- kube-system/coredns-85cb69466-2ff7k:53 to-endpoint FORWARDED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:57009 -> kube-system/coredns-85cb69466-2ff7k:53 L3-L4 REDIRECTED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:57009 -> kube-system/coredns-85cb69466-2ff7k:53 to-proxy FORWARDED (UDP)
Dec 22 23:47:16.766: cilium-test1/dns-client:57009 -> kube-system/coredns-85cb69466-2ff7k:53 dns-request FORWARDED (DNS Query example.com.grickle.local. A)
Dec 22 23:47:16.768: cilium-test1/dns-client:57009 <- kube-system/coredns-85cb69466-2ff7k:53 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy example.com.grickle.local. A))
Dec 22 23:47:16.768: cilium-test1/dns-client:57009 <- kube-system/coredns-85cb69466-2ff7k:53 to-endpoint FORWARDED (UDP)
Dec 22 23:47:16.768: cilium-test1/dns-client:37040 -> kube-system/coredns-85cb69466-2ff7k:53 L3-L4 REDIRECTED (UDP)
Dec 22 23:47:16.768: cilium-test1/dns-client:37040 -> kube-system/coredns-85cb69466-2ff7k:53 to-proxy FORWARDED (UDP)
Dec 22 23:47:16.768: cilium-test1/dns-client:37040 -> kube-system/coredns-85cb69466-2ff7k:53 dns-request FORWARDED (DNS Query example.com. A)
Dec 22 23:47:16.783: cilium-test1/dns-client:37040 <- kube-system/coredns-85cb69466-2ff7k:53 dns-response FORWARDED (DNS Answer "93.184.216.34" TTL: 30 (Proxy example.com. A))
Dec 22 23:47:16.783: cilium-test1/dns-client:37040 <- kube-system/coredns-85cb69466-2ff7k:53 to-endpoint FORWARDED (UDP)
Dec 22 23:47:16.793: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:16.793: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:17.814: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:17.814: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:19.862: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:19.862: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:23.894: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:23.894: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:31.958: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:31.958: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:48.342: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:47:48.342: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:48:20.598: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Dec 22 23:48:20.598: cilium-test1/dns-client:55388 <> example.com:80 Policy denied DROPPED (TCP Flags: SYN)
Raw
z02-SystemInfo.md

Cilium install

cilium install --config enable-host-firewall=true --config devices="eth0"
cilium hubble enable --ui
cilium status
cilium connectivity test

Cilium status

[root@k3s-control ~]# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 3
                  cilium-operator    Running: 1
                  hubble-relay       Running: 1
                  hubble-ui          Running: 1
Cluster Pods:     15/17 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.11.0: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.11.0: 1
                  hubble-relay       quay.io/cilium/hubble-relay:v1.11.0: 1
                  hubble-ui          quay.io/cilium/hubble-ui:v0.8.3: 1
                  hubble-ui          quay.io/cilium/hubble-ui-backend:v0.8.3: 1
                  hubble-ui          docker.io/envoyproxy/envoy:v1.18.2@sha256:e8b37c1d75787dd1e712ff389b0d37337dc8a174a63bed9c34ba73359dc67da7: 1
[root@k3s-control ~]# cilium config view
auto-direct-node-routes                false
bpf-lb-map-max                         65536
bpf-map-dynamic-size-ratio             0.0025
bpf-policy-map-max                     16384
cilium-endpoint-gc-interval            5m0s
cluster-id
cluster-name                           default
cluster-pool-ipv4-cidr                 10.0.0.0/8
cluster-pool-ipv4-mask-size            24
custom-cni-conf                        false
debug                                  false
devices                                eth0
disable-cnp-status-updates             true
enable-auto-protect-node-port-range    true
enable-bpf-clock-probe                 true
enable-bpf-masquerade                  false
enable-endpoint-health-checking        true
enable-health-check-nodeport           true
enable-health-checking                 true
enable-host-firewall                   true
enable-hubble                          true
enable-ipv4                            true
enable-ipv4-masquerade                 true
enable-ipv6                            false
enable-l7-proxy                        true
enable-local-redirect-policy           false
enable-policy                          default
enable-remote-node-identity            true
enable-session-affinity                true
enable-well-known-identities           false
enable-xt-socket-fallback              true
hubble-disable-tls                     false
hubble-listen-address                  :4244
hubble-socket-path                     /var/run/cilium/hubble.sock
hubble-tls-cert-file                   /var/lib/cilium/tls/hubble/server.crt
hubble-tls-client-ca-files             /var/lib/cilium/tls/hubble/client-ca.crt
hubble-tls-key-file                    /var/lib/cilium/tls/hubble/server.key
identity-allocation-mode               crd
install-iptables-rules                 true
install-no-conntrack-iptables-rules    false
ipam                                   cluster-pool
kube-proxy-replacement                 disabled
monitor-aggregation                    medium
monitor-aggregation-flags              all
monitor-aggregation-interval           5s
node-port-bind-protection              true
operator-api-serve-addr                127.0.0.1:9234
preallocate-bpf-maps                   false
sidecar-istio-proxy-image              cilium/istio_proxy
wait-bpf-mount                         true
[root@k3s-control ~]#

K3s info

VERSION:
   v1.22.5+k3s1 (405bf79d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment