When you set up a TCP connection, you associate a set of timers. Some of these timers deal with the keepalive procedure. When the keepalive timer reaches zero, you send your peer a keepalive probe packet with no data in it and the ACK flag turned on. You can do this because of the TCP/IP specifications, as a sort of duplicate ACK, and the remote endpoint will have no arguments, as TCP is a stream-oriented protocol. On the other hand, you will receive a reply from the remote host (which doesn't need to support keepalive at all, just TCP/IP), with no data and the ACK set.
If you receive a reply to your keepalive probe, you can assert that the connection is still up and running without worrying about the user-level implementation. In fact, TCP permits you to handle a stream, not packets, and so a zero-length data packet is not dangerous for the user program.
This procedure is useful because if the other peers lose their connection (for example by rebooting) you will notice that the connection is broken, even if you don't have traffic on it. If the keepalive probes are not replied to by your peer, you can assert that the connection cannot be considered valid and then take the correct action.
Keep alive generates extra network traffic, which can have an impact on routers and firewalls.
- https://cromwell-intl.com/open-source/performance-tuning/tcp.html
- Kernel parameters for TCP are set in /proc/sys/net/core/[rw]mem* and /proc/sys/net/ipv4/tcp*, those apply for both IPv4 and IPv6
- https://www.slashroot.in/linux-network-tcp-performance-tuning-sysctl
- https://community.mellanox.com/s/article/linux-sysctl-tuning
- Time-to-live (TTL) is a value in an Internet Protocol (IP) packet that tells a network router whether or not the packet has been in the network too long and should be discarded
- An IP TTL is set initially by the system sending the packet. It can be set to any value between 1 and 255; different operating systems set different defaults. Each router that receives the packet subtracts at least 1 from the count; if the count remains greater than 0, the router forwards the packet, otherwise it discards it and sends an Internet Control Message Protocol (ICMP) message back to the originating host, which may trigger a resend.
- In IP multicast, the TTL controls the scope or range in which a packet may be forwarded. By convention:
- 0 is restricted to the same host
- 1 is restricted to the same subnet
- 32 is restricted to the same site
- 64 is restricted to the same region
- 128 is restricted to the same continent
- 255 is unrestricted
- Use the traceroute command to diagnose IP routing problems
- !N - network that 10.1.2.4 IP belongs doesn’t exist; ICMP destination unreachable “Network Unreachable” message from this host
- !H - IP network is available but the individual host can't be reached; ICMP destination unreachable "Host unreachable" to the source host.
- !F - Intermediate host which can't deliver this oversized packet returns ICMP Destination unreachable "Fragmentation needed and DF bit set"
- ** ** ** continuously
- There's two reasons for it.
- A timeout. Although usually you'll eventually get replies in one of the rtt positions like 70 * * or something.
- Some routers are configured to block some parts of the ICMP protocol from working (ICMP echo to be precise). This is usually to help prevent DDOS attacks etc. Traceroute depends on ICMP echo working.
- If the process is not started yet, you can start it as a new process and monitor it using the following
strace -f -e trace=network -s 10000 <PROCESS WITH ARGUMENTS>;
- If the process is already started and you know its PID you can use the following
strace -f -e trace=network -s 10000 -p <PID>;
- strace is a very helpful utility that can be used to trace system calls and signals.
- https://bytefreaks.net/gnulinux/how-to-capture-all-network-traffic-of-a-single-process
# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:82:d6:6e brd ff:ff:ff:ff:ff:ff
# ip -br link show
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP 52:54:00:82:d6:6e <BROADCAST,MULTICAST,UP,LOWER_UP>
#ip link set eth0 up
# ip -s link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:82:d6:6e brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
34107919 5808 0 6 0 0
TX: bytes packets errors dropped carrier collsns
434573 4487 0 0 0 0
- check to see if an interface has negotiated the correct speed
- An interface that has negotiated the wrong speed (e.g. a 10Gbps interface that only reports 1Gbps speeds) can be an indicator of a hardware/cabling issue, or a negotiation misconfiguration on one side of the link (e.g., a misconfigured switch port).
# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
- The data link layer is responsible for local network connectivity; essentially, the communication of frames between hosts on the same Layer 2 domain
- Most relevant Layer 2 protocol - ARP (Address Resolution Protocol)
- ARP maps Layer 3 IP addresses to Layer 2 Ethernet MAC addresses.
- When a host tries to contact another host on its local network (such as the default gateway), it likely has the other host’s IP address, but it doesn’t know the other host’s MAC address. ARP solves this issue and figures out the MAC address for us.
- A common problem that could be seen is an ARP entry that won’t populate, particularly for your host’s default gateway
- If your localhost can’t successfully resolve its gateway’s Layer 2 MAC address, then it won’t be able to send any traffic to remote networks. This problem might be caused by having the wrong IP address configured for the gateway, or it may be another issue, such as a misconfigured switch port.
# ip neighbor show
192.168.122.1 dev eth0 lladdr 52:54:00:11:23:84 REACHABLE
- If there was a problem with ARP, then we would see a resolution failure:
# ip neighbor show
192.168.122.1 dev eth0 FAILED
- Another common use of the ip neighbor command involves manipulating the ARP table
- Imagine that your networking team just replaced the upstream router (which is your server’s default gateway). The MAC address may have changed as well since MAC addresses are hardware addresses that are assigned at the factory.
- Linux caches the ARP entry for a period of time, so you may not be able to send traffic to your default gateway until the ARP entry for your gateway times out.
# ip neighbor show
192.168.122.170 dev eth0 lladdr 52:54:00:04:2c:5d REACHABLE
192.168.122.1 dev eth0 lladdr 52:54:00:11:23:84 REACHABLE
# ip neighbor delete 192.168.122.170 dev eth0
# ip neighbor show
192.168.122.1 dev eth0 lladdr 52:54:00:11:23:84 REACHABLE
- Layer 3 involves working with IP addresses
- IP addressing provides hosts with a way to reach other hosts that are outside of their local network
# ip -br address show
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 192.168.122.135/24 fe80::184e:a34d:1d37:441a/64 fe80::c52f:d96e:a4a2:743/64
- The lack of an IP address can be caused by a local misconfiguration, such as an incorrect network interface config file, or it can be caused by problems with DHCP.
- Ping sends an ICMP Echo Request packet to a remote host, and it expects an ICMP Echo Reply in return
# ping www.google.com
PING www.google.com (172.217.165.4) 56(84) bytes of data.
64 bytes from yyz12s06-in-f4.1e100.net (172.217.165.4): icmp_seq=1 ttl=54 time=12.5 ms
64 bytes from yyz12s06-in-f4.1e100.net (172.217.165.4): icmp_seq=2 ttl=54 time=12.6 ms
64 bytes from yyz12s06-in-f4.1e100.net (172.217.165.4): icmp_seq=3 ttl=54 time=12.5 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 12.527/12.567/12.615/0.036 ms
- Many network operators block ICMP packets as a security precaution
- ICMP packets can be rate limited by intermediate network gear, and they shouldn’t be relied upon to provide true representations of application latency.
- Traceroute takes advantage of the Time to Live (TTL) field in IP packets to determine the path that traffic takes to its destination.
- Traceroute will send out one packet at a time, beginning with a TTL of one.
- Since the packet expires in transit, the upstream router sends back an ICMP Time-to-Live Exceeded packet.
- Traceroute then increments the TTL to determine the next hop.
- The resulting output is a list of intermediate routers that a packet traversed on its way to the destination
# traceroute www.google.com
traceroute to www.google.com (172.217.10.36), 30 hops max, 60 byte packets
1 acritelli-laptop (192.168.122.1) 0.103 ms 0.057 ms 0.027 ms
2 192.168.1.1 (192.168.1.1) 5.302 ms 8.024 ms 8.021 ms
3 142.254.218.133 (142.254.218.133) 20.754 ms 25.862 ms 25.826 ms
4 agg58.rochnyei01h.northeast.rr.com (24.58.233.117) 35.770 ms 35.772 ms 35.754 ms
5 agg62.hnrtnyaf02r.northeast.rr.com (24.58.52.46) 25.983 ms 32.833 ms 32.864 ms
6 be28.albynyyf01r.northeast.rr.com (24.58.32.70) 43.963 ms 43.067 ms 43.084 ms
7 bu-ether16.nycmny837aw-bcr00.tbone.rr.com (66.109.6.74) 47.566 ms 32.169 ms 32.995 ms
8 0.ae1.pr0.nyc20.tbone.rr.com (66.109.6.163) 27.277 ms * 0.ae4.pr0.nyc20.tbone.rr.com (66.109.1.35) 32.270 ms
9 ix-ae-6-0.tcore1.n75-new-york.as6453.net (66.110.96.53) 32.224 ms ix-ae-10-0.tcore1.n75-new-york.as6453.net (66.110.96.13) 36.775 ms 36.701 ms
10 72.14.195.232 (72.14.195.232) 32.041 ms 31.935 ms 31.843 ms
11 * * *
12 216.239.62.20 (216.239.62.20) 70.011 ms 172.253.69.220 (172.253.69.220) 83.370 ms lga34s13-in-f4.1e100.net (172.217.10.36) 38.067 ms
- As with ICMP, intermediate routers may filter the packets that traceroute relies on, such as the ICMP Time-to-Live Exceeded message
- Path that traffic takes to and from a destination is not necessarily symmetric, and it’s not always the same.
- Traceroute can mislead you into thinking that your traffic takes a nice, linear path to and from its destination. However, this situation is rarely the case.
- Traffic may follow a different return path, and paths can change dynamically for many reasons.
- While traceroute may provide accurate path representations in small corporate networks, it often isn’t accurate when trying to trace across large networks or the internet.
- Another common issue is likely to be a lack of an upstream gateway for a particular route or a lack of a default route.
- When an IP packet is sent to a different network, it must be sent to a gateway for further processing.
- The gateway should know how to route the packet to its final destination.
- The list of gateways for different routes is stored in a routing table
- routing table can be inspected and manipulated
# ip route show
default via 192.168.122.1 dev eth0 proto dhcp metric 100
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.135 metric 100
- Simple topologies often just have a default gateway configured, represented by the "default" entry at the top of the table.
- A missing or incorrect default gateway is a common issue.
- If our topology is more complex and we require different routes for different networks
# ip route show 10.0.0.0/8
10.0.0.0/8 via 192.168.122.200 dev eth0
- DNS is not a layer 3 protocol.
- Domain Name System (DNS) translates IP addresses into human-readable names
- Most common DNS trouble is the ability to connect to a remote host by IP address but not its hostname
# nslookup www.google.com
Server: 192.168.122.1
Address: 192.168.122.1#53
Non-authoritative answer:
Name: www.google.com
Address: 172.217.3.100
- Host file override issues
- /etc/hosts
# nslookup www.google.com
Server: 192.168.122.1
Address: 192.168.122.1#53
Non-authoritative answer:
Name: www.google.com
Address: 172.217.12.132
# ping -c 1 www.google.com
PING www.google.com (1.2.3.4) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
1.2.3.4 www.google.com
- The transport layer consists of the TCP and UDP protocols, with TCP being a connection-oriented protocol and UDP being connectionless.
- Applications listen on sockets, which consist of an IP address and a port.
- Traffic destined to an IP address on a specific port will be directed to the listening application by the kernel.
TELNET and SSH both are Layer - 7 application layer protocols and both uses TCP at transport layer.
Telnet transfers the data in simple plain text.
SSH uses Encrypted format to send data and also uses a secure channel.
# ss -tunlp4
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 *:68 *:* users:(("dhclient",pid=3167,fd=6))
udp UNCONN 0 0 127.0.0.1:323 *:* users:(("chronyd",pid=2821,fd=1))
tcp LISTEN 0 128 *:22 *:* users:(("sshd",pid=3366,fd=3))
tcp LISTEN 0 100 127.0.0.1:25 *:* users:(("master",pid=3600,fd=13))
ss flags
-t - Show TCP ports.
-u - Show UDP ports.
-n - Do not try to resolve hostnames.
-l - Show only listening ports.
-p - Show the processes that are using a particular socket.
-4 - Show only IPv4 sockets
# telnet database.example.com 3306
Trying 192.168.1.10...
^C
- Here, we can’t get to port 3306 on the remote machine.
- Maybe the application isn’t listening, try using ss on the remote host to check if we have access.
- Another possibility is a host or intermediate firewall that is filtering the traffic. We may need to work with the network team to verify Layer 4 connectivity across the path.
# nc 192.168.122.1 -u 80
test
Ncat: Connection refused.
- TCP and UDP port scanning remote machines.
- OS fingerprinting.
- Determining if remote ports are closed or simply filtered and many other things..