Skip to content

Instantly share code, notes, and snippets.

@bikram20

bikram20/DOKS-Cluster-Network-Analysis-from-Outside

Created May 6, 2021 05:45
Show Gist options
  • Save bikram20/493f685c4fa1f6e453f1db7053acbe4f to your computer and use it in GitHub Desktop.
Save bikram20/493f685c4fa1f6e453f1db7053acbe4f to your computer and use it in GitHub Desktop.
Download ZIP
DOKS worker nodes accessibility from Internet
Raw
DOKS-Cluster-Network-Analysis-from-Outside
# Use "brew install nmap" or any other way to get nmap.
# Have doctl and kubectl configured.
#
bgupta@C02CC1EGMD6M employeeapp % doctl compute droplet list --tag-name 'k8s' --format 'Name'
Name
pool-4fz85fgrm-8rqqw
pool-4fz85fgrm-8yde3
bgupta@C02CC1EGMD6M employeeapp % doctl compute droplet list --tag-name 'k8s' --format 'Name','PublicIPv4'
Name Public IPv4
pool-4fz85fgrm-8rqqw 161.35.114.243
pool-4fz85fgrm-8yde3 134.209.219.51
bgupta@C02CC1EGMD6M employeeapp %
bgupta@C02CC1EGMD6M employeeapp % nmap -F 161.35.114.243
Starting Nmap 7.91 ( https://nmap.org ) at 2021-05-05 20:01 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.08 seconds
bgupta@C02CC1EGMD6M employeeapp % nmap -Pn 161.35.114.243
Host discovery disabled (-Pn). All addresses will be marked 'up' and scan times will be slower.
Starting Nmap 7.91 ( https://nmap.org ) at 2021-05-05 20:02 PDT
Nmap scan report for 161.35.114.243
Host is up.
All 1000 scanned ports on 161.35.114.243 are filtered
Nmap done: 1 IP address (1 host up) scanned in 402.30 seconds
bgupta@C02CC1EGMD6M employeeapp %
bgupta@C02CC1EGMD6M employeeapp % nmap -A 161.35.114.243
Starting Nmap 7.91 ( https://nmap.org ) at 2021-05-05 20:14 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.41 seconds
bgupta@C02CC1EGMD6M employeeapp % nmap -sV 161.35.114.243
Starting Nmap 7.91 ( https://nmap.org ) at 2021-05-05 20:14 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.30 seconds
bgupta@C02CC1EGMD6M employeeapp % nmap -p- 161.35.114.243
Starting Nmap 7.91 ( https://nmap.org ) at 2021-05-05 20:14 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.10 seconds
bgupta@C02CC1EGMD6M employeeapp %
# So all the ports are blocked from outside. The IP addresses are not not exposed to Internet.
# We can check the firewalls that apply to the worker nodes
# Now let us connect to a worker node and inspect the firewall on the kernel
bgupta@C02CC1EGMD6M employeeapp % git clone https://github.com/digitalocean/doks-debug.git
Cloning into 'doks-debug'...
remote: Enumerating objects: 91, done.
remote: Total 91 (delta 0), reused 0 (delta 0), pack-reused 91
Receiving objects: 100% (91/91), 16.66 KiB | 550.00 KiB/s, done.
Resolving deltas: 100% (45/45), done.
bgupta@C02CC1EGMD6M employeeapp % cd doks-debug
bgupta@C02CC1EGMD6M doks-debug % ls
Dockerfile LICENSE README.md k8s script
bgupta@C02CC1EGMD6M doks-debug % kubectl apply -f k8s/daemonset.yaml
daemonset.apps/doks-debug created
bgupta@C02CC1EGMD6M doks-debug % kgpoall
NAMESPACE NAME READY STATUS RESTARTS AGE
default kube-bench-2wn9v 0/1 Completed 0 16d
kube-system cilium-7nzwm 1/1 Running 0 34m
kube-system cilium-fwqdt 1/1 Running 0 22d
kube-system cilium-operator-84bdd6f7b6-4hvm2 1/1 Running 2 22d
kube-system cilium-operator-84bdd6f7b6-nlgth 1/1 Running 2 22d
kube-system coredns-55ff57f948-2lpbt 1/1 Running 0 22d
kube-system coredns-55ff57f948-2x44z 1/1 Running 0 22d
kube-system csi-do-node-b6gdh 2/2 Running 0 34m
kube-system csi-do-node-q6vrt 2/2 Running 0 22d
kube-system do-node-agent-q9jbg 1/1 Running 0 22d
kube-system do-node-agent-zwwdk 1/1 Running 0 34m
kube-system doks-debug-gxwkm 1/1 Running 0 25s
kube-system doks-debug-tlrlc 1/1 Running 0 25s
kube-system kube-proxy-4kw5d 1/1 Running 0 34m
kube-system kube-proxy-nt6p2 1/1 Running 0 22d
bgupta@C02CC1EGMD6M doks-debug %
bgupta@C02CC1EGMD6M doks-debug % kubectl -n kube-system exec -it doks-debug-gxwkm bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@pool-4fz85fgrm-8yde3:~#
root@pool-4fz85fgrm-8yde3:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 134.209.219.51 netmask 255.255.240.0 broadcast 134.209.223.255
inet6 fe80::dccc:aaff:fe8f:adf0 prefixlen 64 scopeid 0x20<link>
ether de:cc:aa:8f:ad:f0 txqueuelen 1000 (Ethernet)
RX packets 10974086 bytes 2489372603 (2.4 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10535154 bytes 1350196311 (1.3 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@pool-4fz85fgrm-8yde3:~#
# From another terminal, ping to the worker node from your laptop
bgupta@C02CC1EGMD6M src % curl icanhazip.com
73.70.228.24
bgupta@C02CC1EGMD6M src % ping 134.209.219.51
PING 134.209.219.51 (134.209.219.51): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
# Come back to the worker node and do a tcpdump for your source IP
root@pool-4fz85fgrm-8yde3:~# tcpdump host 73.70.228.24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@pool-4fz85fgrm-8yde3:~#
# Now let us verify if you can ping one worker node from another. Connect to another worker node and do the ping, while doing a tcpdump.
bgupta@C02CC1EGMD6M src % kubectl -n kube-system exec -it doks-debug-tlrlc bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@pool-4fz85fgrm-8rqqw:~#
root@pool-4fz85fgrm-8rqqw:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 161.35.114.243 netmask 255.255.240.0 broadcast 161.35.127.255
inet6 fe80::bc4d:4dff:fe9c:94df prefixlen 64 scopeid 0x20<link>
ether be:4d:4d:9c:94:df txqueuelen 1000 (Ethernet)
RX packets 14322 bytes 64587231 (64.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8494 bytes 855588 (855.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@pool-4fz85fgrm-8rqqw:~# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.116.0.4 netmask 255.255.240.0 broadcast 10.116.15.255
inet6 fe80::98ce:85ff:fe72:93f3 prefixlen 64 scopeid 0x20<link>
ether 9a:ce:85:72:93:f3 txqueuelen 1000 (Ethernet)
RX packets 31264 bytes 10668107 (10.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 32720 bytes 4007955 (4.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@pool-4fz85fgrm-8rqqw:~#
root@pool-4fz85fgrm-8rqqw:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 161.35.112.1 0.0.0.0 UG 0 0 0 eth0
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.116.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth1
10.244.0.0 10.116.0.3 255.255.255.128 UG 0 0 0 eth1
10.244.1.0 10.244.1.19 255.255.255.128 UG 0 0 0 cilium_host
10.244.1.19 0.0.0.0 255.255.255.255 UH 0 0 0 cilium_host
161.35.112.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
root@pool-4fz85fgrm-8rqqw:~#
root@pool-4fz85fgrm-8rqqw:~# ping 134.209.219.51
PING 134.209.219.51 (134.209.219.51) 56(84) bytes of data.
^C
--- 134.209.219.51 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2033ms
root@pool-4fz85fgrm-8rqqw:~# ping 10.116.0.3
PING 10.116.0.3 (10.116.0.3) 56(84) bytes of data.
64 bytes from 10.116.0.3: icmp_seq=1 ttl=64 time=1.10 ms
^C
--- 10.116.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.102/1.102/1.102/0.000 ms
root@pool-4fz85fgrm-8rqqw:~# ping 134.209.219.51 --help
ping: invalid option -- '-'
Usage: ping [-aAbBdDfhLnOqrRUvV64] [-c count] [-i interval] [-I interface]
[-m mark] [-M pmtudisc_option] [-l preload] [-p pattern] [-Q tos]
[-s packetsize] [-S sndbuf] [-t ttl] [-T timestamp_option]
[-w deadline] [-W timeout] [hop1 ...] destination
Usage: ping -6 [-aAbBdDfhLnOqrRUvV] [-c count] [-i interval] [-I interface]
[-l preload] [-m mark] [-M pmtudisc_option]
[-N nodeinfo_option] [-p pattern] [-Q tclass] [-s packetsize]
[-S sndbuf] [-t ttl] [-T timestamp_option] [-w deadline]
[-W timeout] destination
root@pool-4fz85fgrm-8rqqw:~# ping 134.209.219.51 -I eth1
PING 134.209.219.51 (134.209.219.51) from 10.116.0.4 eth1: 56(84) bytes of data.
^C
--- 134.209.219.51 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2054ms
root@pool-4fz85fgrm-8rqqw:~# ping 10.116.0.3 -I eth1
PING 10.116.0.3 (10.116.0.3) from 10.116.0.4 eth1: 56(84) bytes of data.
64 bytes from 10.116.0.3: icmp_seq=1 ttl=64 time=12.0 ms
^C
--- 10.116.0.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.009/12.009/12.009/0.000 ms
root@pool-4fz85fgrm-8rqqw:~# ping 10.116.0.3 -I eth0
PING 10.116.0.3 (10.116.0.3) from 161.35.114.243 eth0: 56(84) bytes of data.
^C
--- 10.116.0.3 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
root@pool-4fz85fgrm-8rqqw:~# ping 10.116.0.3 -I eth1
PING 10.116.0.3 (10.116.0.3) from 10.116.0.4 eth1: 56(84) bytes of data.
64 bytes from 10.116.0.3: icmp_seq=1 ttl=64 time=3.45 ms
64 bytes from 10.116.0.3: icmp_seq=2 ttl=64 time=1.36 ms
64 bytes from 10.116.0.3: icmp_seq=3 ttl=64 time=0.941 ms
64 bytes from 10.116.0.3: icmp_seq=4 ttl=64 time=1.07 ms
# Notice that all pings pass ONLY to the internal IP of the worker node. Everything directed to external IP is blocked by the DO firewall.
# You can run tcpdump on the worker node to verify that the traffic to the public IP is not even received by the node.
root@pool-4fz85fgrm-8yde3:~# tcpdump -i eth1 host 10.116.0.4 icmp
tcpdump: syntax error in filter expression: syntax error
root@pool-4fz85fgrm-8yde3:~# tcpdump -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
05:42:34.124708 IP 10.116.0.4 > 10.116.0.3: ICMP echo request, id 9367, seq 106, length 64
05:42:34.124788 IP 10.116.0.3 > 10.116.0.4: ICMP echo reply, id 9367, seq 106, length 64
05:42:35.126041 IP 10.116.0.4 > 10.116.0.3: ICMP echo request, id 9367, seq 107, length 64
05:42:35.126107 IP 10.116.0.3 > 10.116.0.4: ICMP echo reply, id 9367, seq 107, length 64
05:42:36.127037 IP 10.116.0.4 > 10.116.0.3: ICMP echo request, id 9367, seq 108, length 64
05:42:36.127097 IP 10.116.0.3 > 10.116.0.4: ICMP echo reply, id 9367, seq 108, length 64
^C
6 packets captured
8 packets received by filter
0 packets dropped by kernel
root@pool-4fz85fgrm-8yde3:~#
# You have the option to harden the worker nodes further through DO firewall.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment