- Prior to version 1.11, Kubernetes used iptables NAT and the conntrack kernel module to track connections. To list all the connections currently being tracked, use the conntrack command:
- To list conntrack-tracked connections to a particular destination address, use the -d flag:
conntrack -L -d 10.32.0.1
-
It's possible your connection tracking table is full and new connections are being dropped. If that's the case you may see messages like the following in your system logs:
$ tail -f /var/log/syslogerror:
Jul 12 15:32:11 worker-528 kernel: nf_conntrack: table full, dropping packet. -
Check the maximum number of connections to tract.
$ sysctl net.netfilter.nf_conntrack_max -
To set a new value, use the -w flag:
sysctl -w net.netfilter.nf_conntrack_max=198000 -
To make this setting permanent, add it to the sysctl.conf file:
$ vi /etc/sysctl.conf . . . net.ipv4.netfilter.ip_conntrack_max = 198000
- Prior to version 1.11, Kubernetes used iptables NAT to implement virtual IP translation and load balancing for Service IPs.
- To dump all iptables rules on a node, use the iptables-save command:
iptables-save - To list just the Kubernetes Service NAT rules, use the iptables command and the -L flag to specify the correct chain:
iptables -t nat -L KUBE-SERVICES
- Check ufw(Uncomplicated firewall) status
$ sudo ufw status- if the status is
Status: active- enable it
$ sudo ufw enable- Check status to verify:
$ sudo ufw status verbose Output: Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), disabled (routed) New profiles: skip
For further more debugging use iptables to debug related articles https://www.dummies.com/computers/operating-systems/linux/how-to-use-netfilter-on-your-linux-system-enabling-a-packet-filtering-firewall/
For kubernetes: A reason for unexplained connection timeouts on Kubernetes/Docker