00-cilium-tcpdump.md

vagrant@k8s1:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i enp0s8 icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes




18:11:10.397859 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
18:11:11.478191 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
18:11:12.509269 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
18:11:14.588899 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
18:11:15.612352 IP6 k8s2 > ff02::1:ff00:2c69: ICMP6, neighbor solicitation, who has f00d::a0f:0:0:2c69, length 32
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel


vagrant@k8s1:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i cilium_host icmp6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_host, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

cilium-bug.md

Node pinging pod IPv6

It happens sometime:

64 bytes from f00d::a0f:0:0:2afe: icmp_seq=72 ttl=63 time=1.70 ms
64 bytes from f00d::a0f:0:0:2afe: icmp_seq=73 ttl=63 time=1.23 ms
64 bytes from f00d::a0f:0:0:2afe: icmp_seq=74 ttl=63 time=0.906 ms
From fd01::c icmp_seq=75 Time exceeded: Hop limit
From fd01::c icmp_seq=76 Time exceeded: Hop limit

On Dst Node

vagrant@k8s1:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i cilium_vxlan icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_vxlan, link-type EN10MB (Ethernet), capture size 262144 bytes
23:19:56.706608 IP6 f00d::a0f:0:0:7239 > f00d::a0f:0:0:2afe: ICMP6, echo request, seq 57, length 64
23:19:56.706896 IP6 f00d::a0f:0:0:2afe > f00d::a0f:0:0:7239: ICMP6, echo reply, seq 57, length 64
23:19:57.708083 IP6 f00d::a0f:0:0:7239 > f00d::a0f:0:0:2afe: ICMP6, echo request, seq 58, length 64

Ping from pod to node: 7239 is a special node that represents the host. It is coming from Node's annotation: io.cilium.network.ipv6-cilium-host: f00d::a0f:0:0:9df5

vagrant@k8s1:~/go/src/github.com/cilium/cilium$ kubectl exec -it client-2 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ # ping fd01::c
PING fd01::c (fd01::c): 56 data bytes
64 bytes from f00d::a0f:0:0:7239: seq=0 ttl=63 time=0.799 ms
64 bytes from f00d::a0f:0:0:7239: seq=1 ttl=63 time=1.237 ms

Pod curling node hanging

The problem is that the node IP is hijected by a wired IP, and return via vxlan:

# Node's VXLAN
vagrant@k8s2:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i cilium_vxlan ip6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_vxlan, link-type EN10MB (Ethernet), capture size 262144 bytes
23:25:29.683732 IP6 k8s1.37128 > f00d::a0f:0:0:1aea.4240: Flags [.], ack 3199782407, win 225, options [nop,nop,TS val 428544 ecr 420643], length 0
23:25:35.445523 IP6 f00d::a0f:0:0:7239.http > f00d::a0f:0:0:2afe.57120: Flags [S.], seq 1227369296, ack 152813607, win 27560, options [mss 1390,sackOK,TS val 340672 ecr 429984,nop,wscale 7], length 0
23:25:35.445901 IP6 f00d::a0f:0:0:2afe.57120 > f00d::a0f:0:0:7239.http: Flags [R], seq 152813607, win 0, length 0
23:25:35.827443 IP6 f00d::a0f:0:0:1aea.4240 > f00d::a0f:0:0:7239.59006: Flags [.], ack 3627811291, win 216, options [nop,nop,TS val 430080 ecr 336991], length 0
23:25:35.827790 IP6 f00d::a0f:0:0:7239.59006 > f00d::a0f:0:0:1aea.4240: Flags [.], ack 1, win 218, options [nop,nop,TS val 340768 ecr 426302], length 0
23:25:36.467462 IP6 f00d::a0f:0:0:7239.http > f00d::a0f:0:0:2afe.57120: Flags [S.], seq 1243336708, ack 152813607, win 27560, options [mss 1390,sackOK,TS val 340928 ecr 430240,nop,wscale 7], length 0

# Node's NATIVE_DEV
vagrant@k8s2:~/go/src/github.com/cilium/cilium$ sudo tcpdump -i enp0s8 ip6 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes
23:27:05.938610 IP6 k8s1.4240 > k8s2.58202: Flags [.], ack 1377621089, win 224, options [nop,nop,TS val 452608 ecr 359521], length 0
23:27:05.938643 IP6 k8s2.58202 > k8s1.4240: Flags [.], ack 1, win 225, options [nop,nop,TS val 363295 ecr 449248], length 0
23:27:06.450783 IP6 fe80::a00:27ff:fe4d:cfe3 > fe80::a00:27ff:feaa:b0b0: ICMP6, neighbor solicitation, who has fe80::a00:27ff:feaa:b0b0, length 32
23:27:06.450875 IP6 fe80::a00:27ff:feaa:b0b0 > fe80::a00:27ff:fe4d:cfe3: ICMP6, neighbor advertisement, tgt is fe80::a00:27ff:feaa:b0b0, length 24
23:27:07.788318 IP6 f00d::a0f:0:0:2afe.57350 > k8s2.http: Flags [S], seq 2901291533, win 27800, options [mss 1390,sackOK,TS val 453070 ecr 0,nop,wscale 7], length 0
23:27:08.825734 IP6 f00d::a0f:0:0:2afe.57350 > k8s2.http: Flags [S], seq 2901291533, win 27800, options [mss 1390,sackOK,TS val 453329 ecr 0,nop,wscale 7], length 0
23:27:10.834969 IP6 f00d::a0f:0:0:2afe.57350 > k8s2.http: Flags [S], seq 2901291533, win 27800, options [mss 1390,sackOK,TS val 453832 ecr 0,nop,wscale 7], length 0