Different Linux utility commands (e.g. top, ps, strace, lsof, netstat, ifconfig, iftop, iptraf, tcpdump, wireshark)

1. linux utilities.md

Start up a container (whichever Linux flavour takes your fancy):

docker run -it ubuntu /bin/bash
docker run -it centos /bin/bash

• top: check what CPU and Memory running processes are utilising
• ps: see what processes are running
• strace: monitor interactions between processes
• lsof: list of open files
• netstat: monitoring network traffic
• ifconfig: configure or review your network interfaces
• iftop: monitors network traffic and displays table of bandwidth usage
• iptraf: monitoring network traffic (more visual than netstat, but not as detailed)
• tcpdump: network packet sniffer
• wireshark/tshark: network packet sniffer and analyser (create pcaps with tcpdump and analyse with tshark)
• telnet: utility for communicating with another host

OSI Model

	Layer	Protocols	Description
7.	Application	HTTP, FTP, SMTP	Window for user app processes
6.	Presentation	JPEG, GIF, MPEG	Format the data to be presented to the Application layer (network translator)
5.	Session	RPC, SQL, NFS	Allow session establishment between processes running on different stations
4.	Transport	TCP, UDP, SPX	Flow control, ensures all messages are delivered error-free, in sequence, no losses or duplications
3.	Network	IP, IPX, ICMP	Routers control operation of subnet, deciding physical path data takes
2.	Data Link	PPP/SLIP	Provides error-free transfer of data over physical layer
1.	Physical	Hub	Physical structure (cables, hubs etc)

ifconfig.md

## ifconfig

The ifconfig command is used to configure a network interface.

For a breakdown of the following ouput see: http://www.aboutlinux.info/2006/11/ifconfig-dissected-and-demystified.html

If you don't want to configure a network interface then running the command without any arguments will display existing network interfaces.

eth0      Link encap:Ethernet  HWaddr 0A:05:1E:A5:6F:FF  
          inet addr:10.6.4.51  Bcast:10.6.7.255  Mask:255.255.248.0
          inet6 addr: fe80::805:1eff:fea5:6fff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9001  Metric:1
          RX packets:8776319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4212889 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10239965628 (9.5 GiB)  TX bytes:10967533931 (10.2 GiB)
          Interrupt:155 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:341240 errors:0 dropped:0 overruns:0 frame:0
          TX packets:341240 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:31612016 (30.1 MiB)  TX bytes:31612016 (30.1 MiB)

We can see above that we have a single Ethernet card (eth0) and a loop back interface (lo)

In newer Linux OS' eth<n> is replaced by p2p<n>

For more information on Network ips and how they're created then read: https://gist.github.com/Integralist/cff468ba808fbca09602

iftop.md

iftop

iftop does for network usage what top(1) does for CPU usage. It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts. Handy for answering the question "why is our ADSL link so slow?"

To install (Ubuntu):

apt-get update
apt-get install iftop

To install (CentOS):

yum install wget libpcap -y
wget http://pkgs.repoforge.org/iftop/iftop-0.17-1.el6.rf.x86_64.rpm
rpm -ivh iftop-0.17-1.el6.rf.x86_64.rpm

• iftop -P: show port numbers

iptraf.md

iptraf

The iptraf tool monitors Inbound and Outbound network traffic passing through a network interface.

We can monitor various connections like TCP, UDP, ICMP, non-IP counts and also Ethernet load information.

To install:

yum install iptraf     # CentOS
apt-get install iptraf # Ubuntu

Of particular interest is the menu Statistical breakdowns... > By TCP/UDP port > eth0 which shows packets for all TCP and UDP ports such as 22 for SSH and 443 for HTTPS.

lsof.md

lsof

An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket)

• lsof -u <user>: filter results by specific user
• lsof -i TCP:22 -n: lists all running processes on port 22 (ssh)
• lsof -i 4: display only IPv4 network files

netstat.md

netstat

netstat (network statistics) is a command-line tool that displays network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interface and network protocol statistics

• netstat -a: show both listening and non-listening sockets (for TCP this means established connections)
• netstat -l: show just listening sockets
• netstat -lt: show just tcp sockets
• netstat -lu: show just udp sockets
• netstat -aep: extend to show the user and also the pids
• netstat -aepn: don't translate host names (e.g. show ip instead)
• netstat -aepT: show host name but don't truncate it
• netstat -ax: show just UNIX domain sockets
• netstat -st: shows summary of connections (useful for identifying TCP connection issues)
• netstat -lc <n_seconds>: continously refreshing every n seconds
• netstat -atepn: nice all-rounder output (see below for example output)

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name   
tcp        0      0 0.0.0.0:8126                0.0.0.0:*                   LISTEN      0          8934       1071/statsd         
tcp        0      0 0.0.0.0:8080                0.0.0.0:*                   LISTEN      498        10087      1355/puma 2.14.0 (t 
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      0          8763       1196/sshd           
tcp        0      0 127.0.0.1:24220             0.0.0.0:*                   LISTEN      497        9296       1307/ruby           
tcp        0      0 10.6.4.51:57228             10.6.31.176:6379            ESTABLISHED 498        218757     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57224             10.6.31.176:6379            ESTABLISHED 498        218743     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:8080              10.6.8.80:48205             ESTABLISHED 498        229190     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57231             10.6.31.176:6379            ESTABLISHED 498        218766     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:57225             10.6.31.176:6379            ESTABLISHED 498        218747     1355/puma 2.14.0 (t 
tcp        0      0 10.6.4.51:8080              10.6.6.76:51764             ESTABLISHED 498        10111      1355/puma 2.14.0 (t 
tcp       53      0 10.6.4.51:56870             54.231.142.40:443           ESTABLISHED 497        229141     1307/ruby           
tcp        1      0 127.0.0.1:34704             127.0.0.1:8080              CLOSE_WAIT  48         226010     20286/httpd

• netstat -r: shows routing table (see below for example output)

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.6.0.0        *               255.255.248.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
default         ip-xx-x-x-x.eu- 0.0.0.0         UG        0 0          0 eth0

ps.md

ps

• ps aux: shows all running processes
• ps axjf: shows ppids (parent process) with nested children pids (j = ppid & f = formatting)
• pstree -aupn <user_to_filter_by>: displays a tree of processes inc. pids and sort by them

a needs to be used with x to give you all processes
u provides additional output such as CPU and MEM
so axu would be more accurate, but I'm more used to aux

strace.md

strace

strace monitors interactions between processes, such as: system calls, signal deliveries, and changes of process state

Install strace (Ubuntu):

apt-get install man -y
apt-get install strace
man strace

Install strace (CentOS):

yum install strace -y

With Docker:
--privileged required for ptrace to be allowed
--security-opt seccomp:unconfined is an alternative

strace <binary_executable>

The end of the strace output will be the output of the specified binary executable.

Filter results using an 'expression':

strace -e trace=open ls

specify multiple system calls with comma-separated list
-e trace=open,read

Attach strace to a running process:

strace -p <process_id>

ps aux to locate process id

Be careful with backgrounded processes. If you attach to a backgrounded process running in the same shell instance as your strace execution, then you'll be locked up.

With Docker you can fix this by executing the following within a new shell:

docker exec -it <container_id> /bin/bash

Then kill -9 <strace_pid|application_pid>

Add a timestamp to your output using -t

Statistical summary -c displays output in a graphical table:

output with -c comes after the binary's output

Redirect and pipe output:

strace php 2>&1 | grep php.ini

tcpdump.md

## tcpdump

tcpdump is a powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface

We can save the output into a pcap file format, that can be viewed by either tcpdump itself or via an open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files

To install:

yum install tcpdump -y     # CentOS
apt-get install tcpdump -y # Ubuntu

Usage

• tcpdump -D: show available network interfaces
• tcpdump -i eth0 -c 5: capture packets from specific network interface (and only set number of packets)
• tcpdump -i eth0 -c 5 -w 0001.pcap: send output to pcap file
• tcpdump -r 0001.pcap: read back out the pcap file content
• tcpdump -n -i eth0 -c 1: converts dns hostnames into ip addresses instead
• tcpdump -n -i eth0 -c 10 tcp: capture only tcp packets
• tcpdump -n -i eth0 -c 10 port 22: capture packets from specific port (e.g. 22)
• tcpdump -i eth0 src <ip>: capture packets from a specific source ip
• tcpdump -i eth0 dst <ip>: capture packets for a specific destination ip
• tcpdump -i eth0 src port 80 or dst port 80: filter all HTTP traffic to or from port 80
• tcpdump -i eth0 -s 0: include contents of each packet (not just the packet header)
• tcpdump -vvv -s 0 -l -n port 53 -XX: watch all DNS traffic (which happens on port 53)

SCP file from AWS remote to local (run on local machine)
scp -v -r <user_name>@<ip>,eu-west-1:<remote_path> <local_path>
Make sure the local directory exists

Flags

• URGENT pointer: used to identify incoming data as 'urgent'. Such incoming segments do not have to wait until the previous segments are consumed by the receiving end but are sent directly and processed immediately
• ACK: used to acknowledge the successful receipt of packets
• Push: exists to ensure that the data is given the priority (used quite frequently at the beginning and end of a data transfer, affecting the way the data is handled at both ends)
• RST: used when a segment arrives that is not intended for the current connection (remote host rejects packet and resets connection)
• SYN: initially sent when establishing the classic 3-way handshake between two hosts (one sent by Host A and one sent back with ACK by Host B)
• FIN: used to tear down the virtual connections created using the previous flag (SYN)

Note: FIN wont necessarily be the last packet sent. For example...

• Host A receives what it believes to be the last data packet
• So Host A sends a FIN to tell Host B it's closing it's connection, along with an ACK to acknowledge Host B's last data packet
• Host B sends an ACK to acknowledge receipt of Host A's FIN
• Host B then sends its own FIN and ACK to close its connection down
• Host A sends a final ACK to acknowledge Host B's last communication was received

telnet.md

telnet

Telnet is a Network Protocol and the tool which uses that protocol (i.e telnet) is also known as Telnet.

The telnet utility is used for interactive communication to a remote/extern host on a given port. Once the connection to the remote host is established, an HTTP request can be send to the host by typing it in the prompt.

The following example shows that the google domain carries out a 302 redirect:

$ telnet www.google.com 80
Trying 87.237.19.30...
Connected to www.google.com.
Escape character is '^]'.
GET #q=cars HTTP/1.1                   

HTTP/1.1 302 Found
Location: http://www.google.co.uk/?gws_rd=cr&ei=k7BjV-GbFOLOgAbd3JbADw#q=cars
Cache-Control: private
Content-Type: text/html; charset=UTF-8
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Date: Fri, 17 Jun 2016 08:10:59 GMT
Server: gws
Content-Length: 268
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=80=R8K0WhuF432ccZzjpchtEPHx-vv1n-9tuoe8P6V2yyNC2h1sd_JB7Q1afFZPo5W9MjvP8UL1ZZ_8UQDHeb3OpGDRNlSNfPkJWqE9JKa9hAJG02wlk7s8eIRy786p7-8U; expires=Sat, 17-Dec-2016 08:10:59 GMT; path=/; domain=.google.com; HttpOnly

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/?gws_rd=cr&amp;ei=k7BjV-GbFOLOgAbd3JbADw#q=cars">here</A>.
</BODY></HTML>
Connection closed by foreign host.

The BBC site gives us a 404 not found:

$ telnet www.bbc.co.uk 80
Trying 212.58.244.66...
Connected to www.bbc.net.uk.
Escape character is '^]'.
GET /news HTTP/1.1

HTTP/1.1 404 Not Found
Content-Type: text/html
Date: Fri, 17 Jun 2016 08:13:29 GMT
Connection: Keep-Alive
Content-Length: 50591

top.md

top

The top command displays processor activity and also displays tasks managed by the kernel in real-time.

• top -u <user>: filter processes by those run by specified user
• top -n 10: stops command running after 10 intervals (otherwise runs forever or until <Ctrl-c>)

The following key strokes should be executed whilst top is running...

• <Shift-o>: display menu so you can change sorting field
• <Shift-p>: changes sorting back to CPU (CPU is already the default behaviour)
• z: toggles on/off ability to have currently running processes highlighted
• c: toggles on/off absolute path for the COMMAND
• d: change interval for screen/data refresh
• k: followed by PID of process you want to kill

wireshark.md

wireshark

Wireshark is a network protocol analyzer.

It lets you see what's happening on your network at a microscopic level.

To install (Mac OS X):

brew options wireshark           # check for options before installing
brew install wireshark --with-qt # install gui version
brew install tshark              # terminal oriented version of Wireshark designed for capturing and displaying packets without need for a gui

For CentOS:

yum install wireshark

With Docker:
--privileged required (otherwise: can't run /usr/sbin/dumpcap: Operation not permitted)
--security-opt seccomp:unconfined is an alternative

Once you have your pcap formatted file (see tcpdump) you can open Wireshark's gui via your terminal by executing the shell command: wireshark

Once open you can use the gui to select "Open Capture File", browse to your pcap file and select it.

Now you can start analysing your network traffic.

You can automate this process by reading in the capture file directly from the shell:

wireshark -r ~/Downloads/tcpdump-tests/0001.pcap

You can also specify the interface to connect to using -i (notice I had to use sudo in order to authorise Wireshark):

sudo wireshark -i en0

To see available interfaces execute: sudo wireshark -D (again you need sudo):

Capture-Message: Capture Interface List ...
Capture-Message: Loading External Capture Interface List ...
1. en0 (Wi-Fi)
2. awdl0
3. bridge0 (Thunderbolt Bridge)
4. en1 (Thunderbolt 1)
5. vboxnet1
6. en2 (Thunderbolt 2)
7. p2p0
8. lo0 (Loopback)

Every time there is (for example) a HTTP request, that might end up being 200 TCP packets, which is difficult to recognize and make sense of manually. But this can be simplified within Wireshark by clicking on Statistics -> Conversations, where it organizes all these disparate packets into TCP sessions.

tshark

When installing Wireshark you'll also get a tshark command, which is a command line version of wireshark.

So you can read in your pcap file like so:

tshark -r ~/Downloads/tcpdump-tests/0001.pcap

This will display clearer formatted analysis than tcpdump -r provides.

tcpdump doesn't know about HTTP or other network protocols. It knows pretty much everything about TCP but it doesn't care what you put inside your TCP packets. tshark on the other hand knows all about what's inside your TCP packets.

sudo tshark -i any \
            -R 'http.request.method == "GET"' \
            -T fields \
            -e http.request.method -e http.request.uri -e ip.dst

The above filters for just packets which have a HTTP GET request in them, and then prints out the request method and the URI for each one.

The way you filter results is by specifying -T and changing to the fields value. From there you can use the -e flag to specify how to filter data. So if you wanted to filter out all the DNS ttls from a tcpdump of just DNS traffic you could use something like:

tshark -r ~/dns-traffic.pcap -T fields -e dns.resp.ttl -e dns.resp.name

Note: if you open the pcap in wireshark, you can find the filter you need by selecting the data manually via the UI and then right-click'ing and selecting "Prepare a Filter > Selected"