Linux and Bash Example

Linux and Bash Example

Machine

uname (short for unix name) is a computer program in Unix and Unix-like computer operating systems that prints the name, version and other details about the current machine and the operating system running on it

uname -a

# This is from a Docker container on Mac
Linux 6fe5c6d1451c 2.6.32-504.23.4.el6.x86_64 #1 SMP Tue Jun 9 20:57:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# This is from Mac
Darwin pladmins-mbp.playerlync.local 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64


# More info
cat /etc/*-release

lsb_release -a

# This is from Ubuntu 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04

hostnamectl

hostnamectl

   Static hostname: my-jump
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 87ed70f204a3c47e682e52da4e23ab99
           Boot ID: 621b8a8d862746699208c6af0af257b2
    Virtualization: kvm
  Operating System: Ubuntu 18.04.5 LTS
            Kernel: Linux 5.4.0-1044-gcp
      Architecture: x86-64

Files

Output files

Redirect output to stdout and file

Resource - StackOverflow - How do I save terminal output to a file?

# Output stream to file; Overwrite file; NOT visible in terminal
SomeCommand > SomeFile.txt 

# Output stream to file; Append to file; NOT visible in terminal
SomeCommand > SomeFile.txt 

#Output Error stream to file; Overwrite file; NOT visible in terminal
command 2> output.txt

#Output error stream to file; Append to file; NOT visible in terminal
command 2>> output.txt

#Output stream standard and error to file; Overwrite file; NOT visible in terminal
command &> output.txt

#Output stream standardand error to file; Append to file; NOT visible in terminal
command &>> output.txt

#Output standard; Overwrite file; visible in terminal
command | tee output.txt

#Output standard and error; Overwrite file; visible in terminal
command |& tee output.txt

#Output standard and error; Append to file; visible in terminal
command |& tee -a output.txt

Clear files contents

cp /dev/null raghu.txt
echo -n "" > raghu.txt
cat /dev/null >raghu.txt
cat /dev/null >raghu.txt

Networking

List Running Process

ps -A | grep dse

Get Open Ports

sudo lsof -i -P -n | grep LISTEN
netstat -pant | grep LISTEN # Linux
netstat -an -ptcp | grep LISTEN # Mac

Get Established Connections

netstat -an | grep ESTABLISHED

get ip address

hostname -a # Linux

Check Remote TCP port

https://www.fosslinux.com/35730/5-ways-to-check-if-a-port-is-open-on-a-remote-linux-pc.htm

curl -v telnet://10.101.33.41:22

nc -z -v -G 5 10.101.36.233 9042 # Mac
netcat -zvw10 10.101.33.41 22 
# -z: zero io mode which is used for scaning
# -v verbose output
# -w10: timeout wait seconds

nmap 10.101.33.41 -p 22

telnet 10.101.33.41 22

echo > /dev/tcp/10.101.33.41/22 && echo "Port is open"

ipconfig & ifconfig

sudo apt install net-tools # Install 

ifconfig
ipconfig getifaddr en0

The arp utility displays and modifies the Internet-to-Ethernet address translation tables used by the address resolution protocol (arp(4)).

apr -a

tpcdump

ifconfig -a # Find loopback adapter
tcpdump -i lo0 # loopback adapter = lo0

sudo tcpdump port 443

DNS

nslookup google.com
dig <todo>

Process

View Process

Linux process management improved with htop

htop

How to List All Running Services Under Systemd in Linux

systemctl list-units --type=service --state=running

Get all process except for the current command

ps aux | grep -v grep |  grep 'Chrome\|USER'

Kill Process

How to Kill a Process in Linux

kill -l # List all signal
kill -1 <pid> # (-HUP) Reload Process
kill -9 <pid> # (-KILL) Kill Process
kill -15 <pid> # (-TERM) gracefully stop a process
pidof <process_name> # list all pids; not on Mac

pkill <process_name>
killall <signal> <process_name>

Get a list of process to kill

echo kill -9  $(ps aux | grep 'cock' | awk '{print $2}')

Get Process ID

pgrep dse

Displays status of jobs in the current shell session

jobs -l

pwdx on Mac - Show current working directory of a process

function pwdx {
  lsof -a -d cwd -p $1 -n -Fn | awk '/^n/ {print substr($0,2)}'
}

File Editing and Searching

view special characters

cat -v

Truncate Line

tput rmam # Truncate
tput sman # Un-Truncate

sed Parse Data using Regular Expressions

Parse Repository, Name, Tag

echo  'xxxxxxxxxx.yyy.ecr.us-east-1.amazonaws.com/sqs-kafka-sink:abcdef' | sed -E 's/([0-9a-zA-Z\.-]*)\/([0-9a-zA-Z-]*)\:([.]*)/\1 - \2 - \3/'

sed - remove Carriage Return & Line Feed

node=$(tr -d '\r\d' <<< $node)

Escape Alias

To escape an alias, prefix command with \

\ls

Execute Multiple Commands

A; B     # Run A and then B, regardless of success of A
A && B   # Run B if and only if A succeeded
A || B   # Run B if and only if A failed
A &      # Run A in background.
(A && B) # Run A in backgroud, follow B in background only if A succeeded

Files and Directories

Creat Multiple directories and assign permissions

sudo mkdir -m 777 -p /Library/WebServer/Documents/Playerlync  \
   /Library/WebServer/Documents/Playerlync/plapi \
   /Library/WebServer/Documents/Playerlync/PLUI \
   /Library/WebServer/Documents/Playerlync/log \
   /Library/WebServer/Documents/Playerlync/config \
   /etc/apache2/vhosts/

Watch

Watch directory or files

watch -n 1 -d "ls -lh /home/ubuntu/node/data/commit-log"

Tail

Use -f to follow

sudo tail -f /var/log/apache2/access.log

List files over a certain size and not in path

Find files and discarding errors

find / -type f -name php.ini 2>/dev/null

Find files file with Regex

find . -regex ".*/kafka.*/.*\.properties"

What does 2>/dev/null mean?
https://askubuntu.com/questions/350208/what-does-2-dev-null-mean

Finding all large files in the root filesystem
https://unix.stackexchange.com/questions/140367/finding-all-large-files-in-the-root-filesystem

How To Find Largest Top 10 Files and Directories On Linux / UNIX / BSD
https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/

find ~/SourceControl -type f -size +1M  -exec 
-sh {} ';' | sort -rh |  grep -v './.git' |  head -n10

Find Multiple Individual File

find /usr/local/lib -type f \( -name '<filename>' -o -name '<filename>' -o\)

Find case-insinsitive files or directorys based on pattern

find / -iname '*http*' -ls 2> /dev/null # No options are need to find Files and Directories & Ignore Errors

Get Directory Size

Stackoverflow - Total size of the contents of all the files in a directory
https://stackoverflow.com/questions/1241801/total-size-of-the-contents-of-all-the-files-in-a-directory

How To Find The Size Of A Directory In Linux https://www.ostechnix.com/find-size-directory-linux/

du -sh

du -h --block-size=M --max-depth=1 node/data/data
1M      node/data/data/solr_admin
1M      node/data/data/system_distributed
1M      node/data/data/dse_leases
154M    node/data/data/keyspace1
1M      node/data/data/system_traces
1M      node/data/data/dse_system
1M      node/data/data/dse_security
1M      node/data/data/system_auth
1M      node/data/data/dse_perf
1M      node/data/data/system_schema
1M      node/data/data/system
1M      node/data/data/dse_system_local
155M    node/data/data

get path or simlink

s -l `which openssl`

Sudo

Difference Between Linux sudo su, su, s, i, bin/bash Privilege Commands

Linux sudo -i privilege command is similar to sudo su – command that opens an "interactive login shell" that is used to switch user ( root or any standard user ) and change the current user home directory to root directory that gives you the root environment, i.e. your ~/.bashrc is ignored.. See the example below:

It is recommend the use of “sudo -i" instead of "sudo -s" for two reasons:

The visual reminder that you are in a ‘root’ session.
The root environment is far less likely to be poisoned with malware, such as a rogue line in .bashrc.

> sudo -i

root@:~# pwd

> pwd
/root

sudo -s command runs a $shell /bin/bash/ with root privileges and gives you the current user’s environment, so your ~/.bashrc is respected.

> sudo -s

> pwd

/home/mikebarlow

Misc

What does ‘sudo usermod -a -G group $USER’ do on Linux?
https://medium.com/@dhananjay4058/what-does-sudo-usermod-a-g-group-user-do-on-linux-b1ab7ffbba9c

How To Add User To Sudoers & Add User To Sudo Group On Ubuntu https://phoenixnap.com/kb/how-to-create-sudo-user-on-ubuntu

sudo usermod -aG docker-compose automaton

Exit Sudo

exit

Declare Bash

#!/bin/bash

pbcopy & pbpaste

pbcopy < file.txt
ps aux | pbcopy

pbpaste > pastetest.txt
pbpaste | grep rcp

Return Column Names with grep

k get jobs --all-namespaces | grep 'NAME\|exp-backup'

redirection w/ heredoc

https://en.wikipedia.org/wiki/Here_document#Unix_shellsq https://linuxize.com/post/bash-heredoc/

Simple example

cat << EOF
The current working directory is: $PWD
You are logged in as: $(whoami)
EOF

Write test to a file with variables

OUT=/tmp/output.txt
 
echo "Starting my script..."
echo "Doing something..."
 
cat <<EOF >$OUT
  Status of backup as on $(date)
  Backing up files $HOME and /etc/
EOF

JQ

Example data

cat file.json | jq .

Get image, where image includes a tag with "abcdef"

# see json above
cat abc-kafka-sink-repo-img-info | jq '.imageDetails[] | select(.imageTags[]? | inside("abcdef"))'

Pull data from AWS and parse with JQ

aws ecr describe-images --repository-name sqs-kafka-sink | jq '.imageDetails[]? | select(.imageTags[]? | inside("abcdef"))'

openssh

openssl s_client -showcerts -connect www.google.com:443 </dev/null

ssh and do a df and return result

ssh -i ~/.ssh/aws-dev.pem admin@${node} -oStrictHostKeyChecking=no /bin/bash -c 'sudo df -hT | grep "ext4\|Filesystem"'

Get drive info

Get AWS Drives

-h : human redable

-T : filesystem type

sudo : use to get all

sudo df -hT | grep /dev
sudo df -hT | grep "/dev\|Filesystem
sudo df -hT | grep "ext4\|Filesystem"

Display Block Devices

-d : Do not print holder devices or slaves

lsblk -df

Block Devices

Exploring Block Devices

df -hT #Get File System info
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  476M     0  476M   0% /dev
tmpfs          tmpfs     493M     0  493M   0% /dev/shm
tmpfs          tmpfs     493M  396K  493M   1% /run
tmpfs          tmpfs     493M     0  493M   0% /sys/fs/cgroup
/dev/xvda1     xfs       8.0G  1.2G  6.9G  15% /
tmpfs          tmpfs      99M     0   99M   0% /run/user/1000

blkid  #Get block divice content type
/dev/xvda1: LABEL="/" UUID="e9c3e7c2-e039-4b2b-9991-955a8c7cd8c0" TYPE="xfs" PARTLABEL="Linux" PARTUUID="a9e8ddc6-e4df-44e6-8c06-a8b069a67bb9"

lsblk #List information about block devices.
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdb    202:16   0   1G  0 disk
xvdf    202:80   0   2G  0 disk

cat /proc/devices #Get the major number and name of the device
Block devices:
  9 md
202 xvd
253 device-mapper
254 mdp
259 blkext

ls -l /sys/fs #List available type of file systems 
total 0
dr-xr-xr-x  2 root root   0 Nov  4 19:16 bpf
drwxr-xr-x 13 root root 340 Nov  4 16:49 cgroup
drwxr-x---  2 root root   0 Nov  4 16:49 pstore
drwxr-xr-x  4 root root   0 Nov  4 19:16 xfs


udevadm info /dev/xvdb #Query sysfs or the udev database
P: /devices/vbd-51792/block/xvdf
N: xvdf
S: sdf
E: DEVLINKS=/dev/sdf
E: DEVNAME=/dev/xvdf
E: DEVPATH=/devices/vbd-51792/block/xvdf
E: DEVTYPE=disk
E: MAJOR=202
E: MINOR=80
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=75893701

file -s /dev/xvda #get file sytem type
/dev/xvda: x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 16777215 sectors, extended partition table (last)\011, code offset 0x63

file -s /dev/xvdb #get file sytem type
/dev/xvdb: data

Managing Block Devices

parted
fdisk -l # Get list of fdisk
cfdisk /dev/xvdb # same as fdisk but with a interface

Add File System

ls -l /sbin/mk* #list filesystem tools
mkfs -t xfs /dev/xvdb
mkdir /newvolume
mount /dev/xvdb /newvolume/
mount

Other Commands

findmnt

Adding a New Device

How To Partition and Format Storage Devices in Linux
https://www.digitalocean.com/community/tutorials/ how-to-partition-and-format-storage-devices-in-linux

How to mount an EBS volume on NVMe based instance types
https://binx.io/blog/2019/01/26/how-to-mount-an-ebs-volume-on-nvme-based-instance-types/

# Run the following 2 commands to identify new Disk on system
parted -l
lsblk

# Create New Partition Standard
parted /dev/xvdf mklabel gpt

# Create New Partition
parted -a opt /dev/xvdf mkpart primary xfs 0% 100% 

# Create File System
mkfs.xfs -L data /dev/xvdf1

#Create Directory
mkdir /mnt/data

# Mount Filesystem
mount -o defaults /dev/xvdf1 /mnt/data

# Add to /etc/fstab
nano /etc/fstab
        /dev/xvdf1 /mnt/data xfs defaults 0 2

# Run
mount -a

Other Steps to Mound a SSD in AWS

Steps :
#mount ssd volumes : 
- sudo mkfs.ext4 /dev/nvme0n1
- sudo mkdir /mnt/nvme0n1
- sudo mount /dev/nvme0n1 /mnt/nvme0n1
- df -hT
- sudo vi /etc/fstab
  - add : /dev/nvme0n1 /mnt/nvme0n1 ext4 noatime,data=writeback,barrier=0,nobh 0 0 
- sudo mount -a
- df -hT

Zip/Tar File

Get Root Directory Name in Tar File

Dir_name=`tar -tzf something-1.3.5a.tar.gz | head -1 | cut -f1 -d"/"`

Docker

# Installing Docker using Convenince Script
# https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# What does ‘sudo usermod -a -G group $USER’ do on Linux?
# https://techoverflow.net/2019/04/30/what-does-sudo-usermod-a-g-docker-user-do-on-linux/
sudo usermod -aG docker <user>

Other

awk

export nodeName=$(k get pod -n infrastructure -o wide | grep image-auto-tagging | awk 'NR==1' |  awk '{sub("\r", "", $7); print $7}')

Alpine - Use apk to install package

apk add --no-cache --wait 10 nano

BarDev/Linux and Bash Commands.md