githubfoam · February 29, 2024 13:35
diff --git a/Linux_Administrator_Daily_Tasks b/Linux_Administrator_Daily_Tasks
 ------------------------------------------------------------------------------------------
 #CLI shortcut keystrokes(linux&MAC)
 Ctrl+L: Clear the screen. This is similar to running the “clear” command.
 Ctrl+C: Interrupt (kill) the current foreground process running in in the terminal. This sends the SIGINT signal to the process
 Ctrl+Z: Suspend the current foreground process running in bash. This sends the SIGTSTP signal to the process. To return the process to the foreground later, use the fg process_name command.
 Ctrl+D: Close the bash shell.This is similar to running the exit command
 Ctrl+L: Clear the screen. This is similar to running the “clear” command.

 Ctrl+S: Stop all output to the screen. This is particularly useful when running commands with a lot of long, verbose output, but you don’t want to stop the command itself with Ctrl+C.
 Ctrl+Q: Resume output to the screen after stopping it with Ctrl+S.

 Ctrl+A or Home: Go to the beginning of the line.
 Ctrl+E or End: Go to the end of the line.
 Alt+B: Go left (back) one word.
 Alt+F: Go right (forward) one word.

 Ctrl + U to clear up to the beginning

 Ctrl + U to recall a command by searching history
 Up Arrow to recall the previous command from history
 Down Arrow to go to next command in the command history

 #multiple virtual consoles
 CTRL + ALT + F1-F8
 -------------------------------------------------------------------------------------------------------------------------------------------------
 #An absolute path is a complete path to a resource, beginning at the filesystem's root
 #The filesystem root is represented as a single forward slash: /

 #Relative path
 /home/damon #if you're already in your home directory
 Music/Rock #the rest of the path relative to where you are now
 -------------------------------------------------------------------------------------------------------------------------------------------------
 #the tilde character "~" represents the current, logged-on user's home directory
 $ cp /etc/ssh/sshd_config ~
 $ cd ~
 #The single dot represents the present working directory, or where you are right now.
 $ sudo cp /etc/ssh/sshd_config .
 #Double dots represent the parent directory, or the directory immediately above the current one in the filesystem
 $ cd ..
 -------------------------------------------------------------------------------------------------------------------------------------------------
 $ bash --version
 ------------------------------------------------------------------------------------------
 /etc #configuration files on the system
 /etc/hosts #maps hostnames to IP addresses
 /etc/skel #files copied to each user's home
 /usr #user by user and supersuser accounts.Stores application programs
 /usr/bin #executables used by users, in user's PATH statement
 /usr/sbin #executables used by superusers
 /usr/local #applications which are not part of linux
 #applications installed after initial linux installation,in user's PATH statement
 #administrative applications installed after initial linux installation
 /usr/local/bin
 /usr/doc #the default location for application documentation is in a directory named for the application
 /var #stores log files, mails and other data
 /var/spool #mail & printing files
 /var/tmp #data that should persist across reboot

 /var/log #log files
 /var/log/anaconda.log - While installing Linux, all installation related messages are stored in this log file
 /var/log/audit/ - This subdirectory contains logs information stored by the Linux audit daemon (auditd)
 /var/log/auth.log - user logins,such as password prompts
 /var/log/boot.log - Contains information that are logged when the system boots
 /var/log/cron - Whenever cron daemon (or anacron) starts a cron job, it logs the information about the cron job in this file
 /var/log/debug #debugging information from the Ubuntu system and applications
 /var/log/daemon.log - display server, SSH sessions, printing services, bluetooth
 /var/log/dmesg - Contains kernel ring buffer. This file is overwritten when the system is rebooted.
 /var/log/faillog -failed user login attempts. ssh(used for remote login), su(to switch users), at, cron (both used for scheduling tasks) use PAM modules for authentication
 /var/log/kern.log - Contains information logged by the kernel. Helpful to troubleshoot a custom-built kernel.
 /var/log/lastlog #last logins
 /var/log/messages #the main system log,the messages that are logged during system startup
 /var/log/sa/ - Contains the daily sar files that are collected by the sysstat package.
 /var/log/samba/ - Contains log information stored by samba, which is used to connect Windows to Linux.
 /var/log/setroubleshoot/ - SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files and logs those information in this log file.
 /var/log/secure #failed login attempts, failed SSH login attempts, successful SSH logins
 /var/log/syslog
 /var/log/user.log - Contains information about all user level logs.

 /var/log/Xorg.x.log - Log messages from the X server to this file.
 /var/log/Xorg.0.log
 /var/log/yum.log - Contains information that are logged when a package is installed using yum

 #binary log files,cannot be read with a normal text editor,special command-line tools are used to display the relevant information in human-readable format
 /var/log/btmp - This file contains information about failed login attempts
 /var/log/utmp - The utmp file allows one to discover information about who is currently using the system.
 /var/log/wtmp - The wtmp file records all logins and logouts.

 #This subdirectory contains additional logs from the mail server. 
 #For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file
 /var/log/mail/ 
 /var/log/maillog - Logs information from the mail server that is running on the system.
 /var/log/mail.log - Logs information from the mail server that is running on the system.

 #- Contains the apache web server access_log and error_log and related virtual hosts logs if set up to log here.
 /var/log/httpd/ 
 /var/log/apache2 
 /var/log/apache2/error.log
 /var/log/mysql.log

 #All printer and printing related log messages
 /var/log/cups 
 /var/log/spooler 

 /bin #binaries run during system startup
 /sbin #administrative binaries run by superusers
 /root #home for superuser
 /home #user home dirs
 /boot #files run by boot loader and kernel
 /dev #peripheral access files
 /proc #virtual dir contains system info
 /tmp #stores temporary files
 /etc/grub.conf or /boot/grub/grub.conf #customize the behavior and appearance of the boot menu
 /dev/shm,/dev/shmfs #also known as tmpfs,df -h,file system which keeps all files in virtual memory,no files created on hard drive
 /etc/passwd #UID/GID stored, UID/GUID zero(0) for root, all UID/GID <1000 privileged access, all other users UID/GID>1000

 #list group members
 /etc/group #secondary group memberships, 
 awk -F':' '/sudo/{print $4}' /etc/group 
 grep '^sudo' /etc/group
 grep -w sudo /etc/group
 ------------------------------------------------------------------------------------------
 #Bard

 The command [ -f /var/run/reboot-required ] && sudo reboot -f first checks if the file /var/run/reboot-required exists. If it does, the command then runs the sudo reboot -f command, which forcefully reboots the system.

 The file /var/run/reboot-required is created by some Linux distributions when a system update requires a reboot. This is to ensure that the system is properly updated before it is restarted.

 The sudo reboot -f command forcefully reboots the system, even if there are unsaved changes in open files. This is useful if the system is unresponsive or if you need to reboot it immediately.

 Some of the best uses cases for the command [ -f /var/run/reboot-required ] && sudo reboot -f are:

    After a system update that requires a reboot.
    If the system is unresponsive.
    If you need to reboot the system immediately.

 It is important to note that forcefully rebooting the system can sometimes cause data loss. Therefore, you should only use this command if you are sure that there are no unsaved changes in open files.
 ------------------------------------------------------------------------------------------
 #ChatGPT

 This command is used in a Linux shell environment and is composed of two parts connected by the && operator. The overall purpose of this command is to check if a specific file exists and, if it does, initiate a system reboot with administrative privileges. Let's break down each part of the command:

    [ -f /var/run/reboot-required ]: This is a conditional test using the [ (test) command. The -f flag checks if the specified file (/var/run/reboot-required) exists. This file is often used on Ubuntu-based systems to indicate that a system reboot is required due to software updates or configuration changes.

    &&: This is the logical "AND" operator. It's used to execute the command on the right only if the command on the left (the test in this case) evaluates as true or successful.

    sudo reboot -f: If the previous test [ -f /var/run/reboot-required ] is successful (meaning the file exists), this command is executed. The sudo command is used to execute the subsequent command with superuser (root) privileges. reboot is a command used to restart the system, and the -f flag forces an immediate reboot without further confirmation.

 Best Use Cases:

    Automated Reboot after Updates: This command is commonly used in automated scripts or scheduled tasks, particularly after software updates have been applied to a Linux system. If the file /var/run/reboot-required exists, it suggests that important updates have been installed that require a system restart for them to take effect. By using this command, you can ensure that the system automatically reboots when necessary after updates.

    Post-Configuration Reboot: In situations where configuration changes are applied that necessitate a reboot, this command can be used to automatically restart the system. For example, if you've made changes that affect kernel modules or critical system services, an automated reboot might be necessary to apply those changes.

    Server Maintenance and Deployment Scripts: When performing maintenance tasks on servers or during deployment procedures, administrators might use this command to ensure that a system is properly restarted after specific actions are taken.

    Scripted System Checks: This command can be incorporated into system health-check scripts that are run periodically to ensure the system is in a proper state. If the script detects that a reboot is required, it can automatically initiate one.

 It's important to exercise caution when using commands that trigger a system reboot, especially in automated scenarios. Rebooting a system without proper planning can lead to unexpected downtime and disruption of services. Always thoroughly test any scripts or commands in a controlled environment before deploying them in production systems.

 ------------------------------------------------------------------------------------------
 "&&" 
 #where the second command is executed only if the exit status of the preceding command is 0 (zero exit code)
 #The right side of && is evaluated if the exit status of the left side is zero (i.e. true)

 "||" 
 #If the exit status of the first command is not 0 (non-zero exit code), then execute the second command
 #The right side of || is evaluated if the left side exit status is non-zero (i.e. false)

 #Opposite to &&, the statement after the || operator will only be executed if the exit status of the test command is false
 [ -f /etc/resolv.conf ] && echo "$FILE exist." || echo "$FILE does not exist."

 ------------------------------------------------------------------------------------------
 #Syntax for Command Substitution
 #The old-style uses backticks (also called backquotes) to wrap the command being substituted
 #The new style begins with a dollar sign and wraps the rest of the command in parenthesis
 $ rpm -ql $(rpm -qa | grep httpd)

 $(command)  #Command Substitution 
 (list) #Group commands in a subshell: ( )
 { list; } #Group commands in the current shell: { }
 [[ expression ]]  #Test - return the binary result of an expression: [[ ]]

 command substitution; output from pwd works as the argument for echo command
 $ echo `pwd`

 Command Substitution
 # `backquotes` also known as `backticks`
 KERNEL_VERSION=`uname -r`
 #(parentheses)
 KERNEL_VERSION=$( uname -r )
 $ echo $KERNEL_VERSION
 4.15.0-29-generic

 #Command substitution in Bash
 #execute a command and substitute it with its standard output,command executes within a subshell

 $ var_date=$(date) && echo $var_date
 $ due_date="01-01"
 $ echo "Status as of $(date +%m-%d-%Y) : The delivery is due on ${due_date}-2022"

 echo "${myvar:-bash}"    #check the variable, $myvar is set or unset. If $myvar is unset, then the string ‘bash’ will print
 echo "${myvar:=bash}"	 #set the value, ‘bash’ to $myvar and print ‘bash’ to the terminal if $myvar is unset
 echo "${myvar:+python}"  #print, ‘python’ to the terminal if $myvar is set before

 $ mystr="Bangladesh"
 $ echo "${mystr:0:6}" #six characters from $mystr starting from position 0 to 6
 $ echo "${mystr:6}" #all characters from $mystr, starting from position 6 to the end	
 $ echo "${#mystr}" #count and print the total number of characters of $mystr

 The process substitution >(command) will be replaced by a file name.
 ------------------------------------------------------------------------------------------
 { Commands; } : Commands execute in current shell
 ( Commands ) : Commands will execute in a subshell

 $ VAR="1"; { VAR="2"; echo "Inside group: VAR=$VAR"; }; echo "Outside: VAR=$VAR"
 Inside group: VAR=2
 Outside: VAR=2

 #the variable is changed in a subshell. Therefore, the changes will not affect the outer shell
 $ VAR="1"; ( VAR="2"; echo "Inside group: VAR=$VAR" ); echo "Outside: VAR=$VAR"
 Inside group: VAR=2
 Outside: VAR=1

 $ pwd; { cd /etc; pwd; }; pwd #current shell
 $ pwd; ( cd /etc; pwd ); pwd #subshell

 ------------------------------------------------------------------------------------------
 #Placing a list of commands between curly braces causes the list to be executed in the current shell context. 
 #No subshell is created.

 $ { ls && v=3; } > tmp
 $ echo "$v"
 3

 #pipe creates a new subshell
 $ { ls a* && v=3; } | grep my
 $ echo "$v" #does not print variable v



 touch abc{1,2,3,4}
 echo ${month[3]}
 echo {10..0}
 mkdir /some/dir/{a,b,c,d}
 mkdir -p -v /home/jon/tmp/{dir1,anotherdir,similardir} #Apply a command on files with different names

 ------------------------------------------------------------------------------------------
 #Parameter Expansion
 #specify a variable within {} to protect it against expansion. 
 #useful when the characters immediately following it can be interpreted as part of the variable name.
 $ price=5
 $ echo "${price}USD"
 ------------------------------------------------------------------------------------------
 # avoid dot slash "./" when running executable scripts, binary file

 #OPTION 1
 export PATH=$PATH:/path/to/directory

 #OPTION 2
 ln -s /path_to_script/myscript /usr/bin/myscript 

 #OPTION 3
 bash -c /Users/you/myscript.sh

 #OPTION 4 , add the line below into  "~/.bashrc." 
 export PATH=/path_to_folder_containing_executable/:$PATH
 ------------------------------------------------------------------------------------------
    #Hosts File
    Windows 10 - "C:\Windows\System32\drivers\etc\hosts"
    Linux - "/etc/hosts"
    Mac OS X - "/private/etc/hosts"
 -------------------------------------------------------------------------------------------------------------------- 
 #Reading file content from command line
 $ while read line; do echo $line; done < company.txt
 --------------------------------------------------------------------------------------------------------------------
 mount -t tmpfs tmpfs /mnt/tmp #in-memory filesystem,tmpfs is a temporary filesystem that only exists in RAM
 --------------------------------------------------------------------------------------------------------------------
 #which is an external binary, located at /usr/bin/which which steps through the $PATH environment variable and checks for the existence of a file
 which modprobe
 #command is built in to shell, with the -v option how shell invokes the command specified as its option
 command -v modprobe
 --------------------------------------------------------------------------------------------------------------------
 #ls -ai command
 
 $ ls ..  # double dot(..) represents the parent directory
 $ sudo rm ..
 rm: cannot remove '..': Is a directory
 $ sudo rm -rf ..
 rm: refusing to remove '.' or '..' directory: skipping '..'

 $ ls .  #single dot(.) represents the current directory
 $ sudo rm .
 rm: cannot remove '.': Is a directory
 $ sudo rm -rf .
 rm: refusing to remove '.' or '..' directory: skipping '.'
 --------------------------------------------------------------------------------------------------------------------
 #list all failed SSH logins
 grep "Failed password" /var/log/auth.log
 egrep "Failed|Failure" /var/log/auth.log
 cat /var/log/auth.log | grep "Failed password"
 grep "Failed password" /var/log/auth.log | awk '{print $11}' | uniq -c | sort -nr # the number of failed attempts of each IP address,
 --------------------------------------------------------------------------------------------------------------------
 tar -czf - ./Documents/ | (pv -p --timer --rate --bytes > backup.tgz) #Monitor tar progress
 pv -p history.log | wc #Count number of lines, words, bytes
 pv -p /etc/hosts | wc
 pv history.log | zip>$HOME/Documents/history.zip #history.log  and show progress
 pv origin-cdn.cyberciti.org_access.log > /tmp/origin-cdn-access.log #copy a file and show progress
 pv origin-cdn.cyberciti.org_access.log > /dev/null #show progress
 pv -cN rawlogfile origin-cdn.cyberciti.org_access.log | gzip | pv -cN gziplogfile > access.log.gz #see progress of both pipes

 (pv -n backup.tar.gz | tar xzf - -C path/to/data ) 2>&1 | dialog --gauge "Running tar, please wait..." 10 70 0 #extract tar ball and show progress using the dialog command
 tar -czf - ./Documents/ | (pv -n > backup.tgz) 2>&1 | dialog --gauge "Progress" 10 70

 nc -l -v -w 30 -p 2000 > /tmp/data.bin #create a network port 2000
 nc -l -v -w 30 -p 2000 > /tmp/data.bin #Open another terminal
 --------------------------------------------------------------------------------------------------------------------
 timeout 10s tail -f /var/log/pacman.log #terminate after 10 seconds
 timeout 3.2s dmesg -w #display all messages from the kernel ring buffer, only 3.2 seconds
 timeout 3m ping 127.0.0.1 #ping command after 3 minutes
 timeout 2d ping 127.0.0.1 #ping command after 2 days
 timeout --foreground 2m test.sh #By default runs in background, run it foreground

 timeout -k 20 10 tail -f /var/log/pacman.log #if the command is still running even after the time out, send a kill signal
 timeout -s 9 3s ping 127.0.0.1 #use 9 as SIGKILL
 kill -l #list all acceptable signal
 timeout -k 5s 3m sh test.sh #let the script run for 3 minutes, if it does not exit, kill after 5 seconds
 --------------------------------------------------------------------------------------------------------------------
 /home/user01/test.file can also be denoted by ~/test.file #The tilde (~) is a Linux "shortcut" to denote a user's home directory.
 cd ~ #change into user's home directory
 --------------------------------------------------------------------------------------------------------------------
 w #show who is logged on and what they are doing.
 w user #print info for a specific user
 w -i #display IP address instead of hostname for from field.
 w -o #print blank space for idle times less than one minute
 --------------------------------------------------------------------------------------------------------------------
 # stops after 50secs, stdout seen
 skaffold dev & sleep 50s; kill $!
 --------------------------------------------------------------------------------------------------------------------
 $ ssh [email protected] 'bash -s' < script.sh #Execute the local script.sh on the remote server ssh
 $ ssh [email protected] 'uptime; df -h'
 $ ssh [email protected] 'free -m | cat /proc/loadavg'

 $ ssh [email protected] << EOF
 uname -a
 lscpu  | grep "^CPU(s)"
 grep -i memtotal /proc/meminfo
 EOF
 --------------------------------------------------------------------------------------------------------------------
 #setting system locale to en_US.utf8
 localectl set-locale LC_CTYPE=en_US.utf8
 localectl status
 ----------------------------------------------------------------------------------------------------
 ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1 # print IP with a given interface
 ip -o -6 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1
 ifconfig | awk '/192.168.18.84/ {print $1}' RS="\n\n" # print interface with a given IP
 ----------------------------------------------------------------------------------------------------
 #-n     do not output the trailing newline
 echo -n $PASSWORD | faas-cli login --username admin --password-stdin
 ----------------------------------------------------------------------------------------------------
 cat verifyURL.sh
 #!/bin/bash

 url="https://gist.github.com/githubfoam/d313d580a92d84123b841ebbd4f255a6"

 if wget $url >/dev/null 2>&1 ; then
        echo "Url : $url ...is online"
 else
        echo "Url : $url ...is not online"
 fi

 #if link exists
 #--head avoid downloading the file contents
 #--fail make the exit status nonzero on a failed request
 #--silent avoid status or errors 
 url="https://www.katacoda.com/courses/kubernetes/launch-single-node-cluster"
 if curl --output /dev/null --silent --head --fail "$url"; then
  echo "URL exists: $url"
 else
  echo "URL does not exist: $url"
 fi

 time curl -I http://mydomain.com | grep HTTP # get the header for the page, and time the process
 curl -I "www.google.com" 2>&1 | awk '/HTTP\// {print $2}' # see only the HTTP status code
 curl -I "https://www.google.com" 2>&1 | grep -w "200\|301" # see if a given website is up or down
 curl -w "\n" http://localhost:8080/hello # avoid your terminal printing a '%' or put both result and next command prompt on the same line
 lynx -head -dump http://www.google.com #Check Whether a Website is up or down
 lynx -head -dump http://www.google.com 2>&1 | awk '/HTTP\// {print $2}' # see only the HTTP status code

 #SSL certificate problem 
 curl -kfsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
 curl https://curl.se/ca/cacert.pem -o /etc/pki/ca-trust/source/anchors/curl-cacert-updated.pem && update-ca-trust
 wget --no-check-certificate http://curl.haxx.se/ca/cacert.pem
 ----------------------------------------------------------------------------------------------------
 $ time ./a.out
 <output from code>

 real    0m5.279s #real is the wall-clock time,Elapsed real time, real time, wall-clock time, or wall time is the actual time taken from the start of a computer program to the end. In other words, it is the difference between the time at which a task finishes and the time at which the task started. 
 user    0m1.915s #user is the time spent executing the user’s program,
 sys     0m0.006s #sys is the time spent on system tasks required by the program
 ----------------------------------------------------------------------------------------------------
 #linux benchmark
 sysbench --test=cpu run
  sysbench --test=cpu help
 sysbench --test=cpu --cpu-max-prime=20000 run
 sysbench --test=memory run
 sysbench --test=memory help
 sysbench --test=fileio help
 sysbench --test=fileio --file-test-mode=seqwr run
 sysbench --test=fileio --file-total-size=100G cleanup
  ----------------------------------------------------------------------------------------------------
 # generate load
 $ yes > /dev/null &

 # maximum number of processes available to a single user.
 $ ulimit -u
 # The limit is set with the -S option
 $ ulimit -S -u 500

 #inspect the load figures on the server before and after the stress test
 uptime
 sudo stress --cpu 100 --timeout 300
 uptime

 $ sudo (yum/apt-get) install stress
 ## Stress using CPU-bound task
 stress -c 4
 ## Stress using IO-bound task 
 stress -i 2
 #  a load average of four is imposed on the system by specifying two CPU-bound processes, one I/O-bound process, and one memory allocator process
 stress -c 2 -i 1 -m 1 --vm-bytes 128M -t 10s

 $ sudo apt-get install stress-ng
 stress-ng --cpu 4 --io 4 --vm 1 --vm-bytes 1G --timeout 60s --metrics-brief
 stress-ng --hdd 5 --hdd-ops 100000
 stress --cpu 4 --io 3 --vm 2 --vm-bytes 256M --timeout 35s

 # The restriction can be made permanent by configuring the nproc value
 /etc/security/limits.conf
 # initiate a fork bomb
 ":(){ :|:& };:"
 # run ./$0& twice
 #!/bin/sh
 ./$0&
 ./$0&
 ----------------------------------------------------------------------------------------------------
 # Listing all currently known events:
 perf list
 # CPU counter statistics for the specified command:
 perf stat command
 # CPU counter statistics for the specified PID, until Ctrl-C:
 perf stat -p PID
 # Sample on-CPU functions for the specified command, at 99 Hertz:
 perf record -F 99 command
 # Trace all block device (disk I/O) requests with stack traces, until Ctrl-C:
 perf record -e block:block_rq_insert -ag
 # Add a tracepoint for the kernel tcp_sendmsg() function entry ("--add" is optional):
 perf probe --add tcp_sendmsg
 # Trace system calls by process, showing a summary refreshing every 2 seconds:
 perf top -e raw_syscalls:sys_enter -ns comm
 # Show perf.data with a column for sample count:
 perf report -n

 ----------------------------------------------------------------------------------------------------
 #ls command
 ltrace -c ls
 ltrace -p <PID>
 ltrace -l /lib/libselinux.so.1 id -Z #execute the id -Z command and show the calls made to the libselinux.so module
 ltrace -o foobar.log ./foobar #edirect output of ltrace to a file
 ltrace -e malloc ./foobar #filter and display only calls to a certain library function

 #ls command
 strace ls
 strace -v #-v verbose option that can provide additional information on each system call
 strace -s 80 -f ./program	#print the first 80 characters of every string
 strace -i ls # print instruction pointer at the time of system call
 strace -r ls # display a relative timestamp upon entry to each system call
 strace -t ls #each line in strace output to start with clock time
 strace -T ls #show time spent in system calls
 strace -c ls #print a summary
 strace -p 3569 #If a process is already running, you can trace it by simply passing its PID
 strace -p `pidof rsyslogd`
 strace -p $(pgrep rsyslogd) #monitor process without knowing its PID, but name
 #if there are multiple processes to be traced at once (e.g. all instances of an running apache httpd)
 strace $( pgrep httpd | sed 's/^/-p/' ) 
 strace -c -p 3569 Summary of Linux Process
 strace -c ls Counting number of sys calls
 strace -o output.txt ls
 strace -e trace=network -o OUTFILE php -q test2.php # write the output of strace to a file or redirect the output
 strace -e trace=network php -q test2.php 2> test2debug #redirect the output
 strace -e trace=open,stat,read,write ls
 strace -e trace=mprotect,brk ifconfig eth0 # trace mprotect or brk system calls
 strace -e trace=network ifconfig eth0 #Trace all the network related system calls
 strace -e trace=network #Monitoring the network
 strace -e trace=memory Monitoring memory calls
 strace -e open ls  display only a specific system call, use the strace -e option
 strace -f -eopen /usr/sbin/sshd 2>&1 | grep ssh	shows the three config files that OpenSSH’s sshd reads as it starts,strace sends its output to STDERR by default
 strace -e trace=file -p 1234 #See all file activity,Monitoring file activity
 strace -e trace=desc -p 1234
 strace -P /etc/cups -p 2261 #track specific paths, use 1 or more times the -P parameter, following by the path
 strace -f -o strace_acroread.txt acroread #follow system calls if a process fork
 strace -q -e trace=process df -h trace all system calls involving process management.
 strace -q  -e trace=file df -h trace all system calls that take a filename as an argument
 strace -f -e execve ./script.sh #check what commands are exactly being executed by a script by using strace
 strace -f -e execve bash x.sh

 $ strace e execve bash -c true
 $ strace -ve execve bash -c true
 $ strace -e execve bash -c /bin/true

 $ strace -o OUT -ff -e execve bash -c "/bin/true ; /bin/false"
 $ grep execve OUT*
 OUT.29328:execve("/usr/bin/bash", ["bash", "-c", "/bin/true"], 0x7ffc75ace798 /* 25 vars */) = 0
 OUT.29328:execve("/bin/true", ["/bin/true"], 0x55bf673522c0 /* 25 vars */) = 0
 OUT.29336:execve("/usr/bin/bash", ["bash", "-c", "/bin/true ; /bin/false"], 0x7ffe1b316638 /* 25 vars */) = 0
 OUT.29337:execve("/bin/true", ["/bin/true"], 0x55aba17c92c0 /* 25 vars */) = 0
 OUT.29338:execve("/bin/false", ["/bin/false"], 0x55aba17c92c0 /* 25 vars */) = 0

 # Under Linux, fork is a special case of the more general clone system call, which you observed in the strace log.
 #The child runs a part of the shell script. The child process is called a subshell.
 strace -f -o bash-mystery-1.strace bash -c 'v=15; (echo $v)'
 strace -f -o bash-mystery-2.strace bash -c 'v=15; bash x.sh'
 man 2 clone #create a child process
 grep clone bash-mystery-2.strace # filter child process

 strace -e open,read,write cat /etc/HOSTNAME
 strace -e open,read,write cat /etc/HOSTNAME > /dev/null
 strace -e file cat /etc/HOSTNAME

 #Execute Strace on a Running Linux Process Using Option -p
 ps -C firefox-bin #PID 126
 sudo strace -p 126 -o firefox_trace.txt #display the following error when your user id does not match the user id of the given process.
 pidof sshd #PID 126
 strace -p 126

 #sleep.sh, endless loop
 #! /bin/bash
 while :
 do
 sleep 10 &
 echo "Sleeping for 4 seconds.."
 sleep 4
 done

 $ sh sleep.sh & # run in the background
 $ pstree -p #see sleep.sh child/parent processes
 $ pgrep sleep | sed 's/^/-p/'
 $ pidof sleep
 $  sudo strace -c -fp PID # attach to parent process of the sleep.sh
 $ strace -c -fp $( pgrep sleep | sed 's/^/-p/' ) # another terminal, monitor multipe=le child processes of sleep.sh
 $ strace $( pgrep sleep | sed 's/^/-p/' ) # another terminal, monitor multipe=le child processes of sleep.sh

 man 3 stat #access the documentation. stat is the system call that gets a file's status
 man 2 execve

 $ grep openat trace.log
 ----------------------------------------------------------------------------------------------------
 PATH=/data/myscripts:$PATH #add directory /data/myscripts to the beginning of the $PATH environment variable
 PATH=$PATH:/data/myscripts #add that directory to the end of the path

 echo 'export PATH=$PATH:/new/directory' >> ~/.zshrc
 source ~/.zshr
 ----------------------------------------------------------------------------------------------------
 echo "$(pwd)"
 #interpret backslash escapes 
 echo -e "Tecmint \nis \na \ncommunity \nof \nLinux \nNerds" 
 echo -e "Tecmint \bis \ba \bcommunity \bof \bLinux \bNerds" 
 echo -e "Tecmint \tis \ta \tcommunity \tof \tLinux \tNerds" 
 echo -e "\vTecmint \vis \va \vcommunity \vof \vLinux \vNerds" 
 echo -e "\n\vTecmint \n\vis \n\va \n\vcommunity \n\vof \n\vLinux \n\vNerds" 

 echo -e "Geeks \bfor \bGeeks"  -e here enables the interpretation of backslash escapes,\b : it removes all the spaces in between the text
 echo -e "Geeks \cfor Geeks" \c : suppress trailing new line with backspace interpretor ‘-e‘ to continue without emitting new line.
 echo -e "Geeks \nfor \nGeeks"  \n : this option creates new line from where it is used.
 echo -e "Geeks \tfor \tGeeks" \t : this option is used to create horizontal tab spaces.
 echo -e "Geeks \vfor \vGeeks \v : this option is used to create vertical tab spaces.
 echo -e "# SNMP version 2c community\nrocommunity monsvronly 192.168.58.8" >> /etc/snmp/snmpd.conf
 echo * #print all files/folders, similar to ls command 
 ----------------------------------------------------------------------------------------------------
 #sh calls the program sh as interpreter and the -c flag means execute the following command as interpreted by this program
 #sh -c spawns a non-login, non-interactive session of sh (dash in Ubuntu). 

 $ sudo sh -c "echo 0" #In Ubuntu, sh is usually symlinked to /bin/dash, meaning that if you execute a command with sh -c the dash shell will be used to execute the command instead of bash
 $ readlink -e $(which sh)
 /usr/bin/dash
 sudo sh -c 'ls -hal /root/ > /root/test.out' #The redirection of the output is performed by sudo.
 sudo sh -c "echo foo > ~root/out.txt"
 sudo echo "foo" | sudo dd of=/root/test2.out
 sudo su -c 'echo "foo" > ~root/test4.out'
 sudo su -c 'echo "foobarr" | tee  ~root/test5.out'
 sudo su -c 'echo "append foobarr " | tee  -a ~root/test5.out'
 sudo ls /root/out.txt

 $ sudo bash -c "echo 0"
 $ readlink -e $(which bash)
 /usr/bin/bash

 #the features that are specific to interactive shells only (by default), e.g. history expansion, source-ing of ~/.bashrc and /etc/bash.bashrc etc will not be available in this session as it is non-interactive
 #simulate an interactive sessions behavior (almost), by using the -i option
 sh -ic 'ls -hal /root/ > /root/test.out'
 #the features that are specific to login shells only (by default) e.g. source-ing of ~/.profile (given ~/.bash_profile and ~/.bash_login do not exist) and /etc/profile will not be done as the shell is a non-login shell
 #simulate login-shells behavior using the -l option
 sh -lc 'ls -hal /root/ > /root/test.out'
 #simulate both login and interactive sessions
 sh -lic 'ls -hal /root/ > /root/test.out'
 ----------------------------------------------------------------------------------------------------
 brew install <formula> #install a package (or Formula in Homebrew vocabulary) 
 brew uninstall <formula>
 brew upgrade <formula>
 brew update
 brew list --versions
 brew search <formula>
 brew info <formula>
 brew cleanup --dry-run #see what formulas Homebrew would delete without actually deleting

 #Homebrew-Cask extends Homebrew and allows you to install large binary files via a command-line tool
 brew tap caskroom/cask #make Cask available by adding it as a tap

 #https://brew.sh/
 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" #Install Homebrew
 ----------------------------------------------------------------------------------------------------
 bash -c "$(curl -sL https://raw.githubusercontent.com/ilikenwf/apt-fast/master/quick-install.sh)" #remote script install
 curl https://raw.githubusercontent.com/golang/dep/v0.5.1/install.sh | sh #remote script install
 ----------------------------------------------------------------------------------------------------

 ln -s $PWD/istioctl /usr/local/bin/istioctl #create symbolic link
 ----------------------------------------------------------------------------------------------------
 hwclock --utc --systohc # set the hardware clock to match system (software) clock on a Linux-only computer, The --utc option tells it to use UTC, which is appropriate for a Linux-only system, --systohc sets the hardware clock based on the current value of the software clock
 ----------------------------------------------------------------------------------------------------
 logger shutting down to add network card #create a log file entry noting that you’re manually shutting down the system to add a new network card prior to using shutdown, The logger utility can be used to create a one-time log fle entry that you specify
 ----------------------------------------------------------------------------------------------------
 rsyslogd -v #Check if rsyslog is installed
 rsyslogd -f /etc/rsyslog.conf -N1 #Check Rsyslog Configuration for Errors

 semanage port -l| grep syslog #view the current SELinux ports settings for rsyslog
 semanage port -a -t syslogd_port_t -p udp 541 #add a UDP port 541 to SELinux

 #Verifying the Configuration

 #On the client server
 logger "Test message from the system `hostname`" 
 nc -u 192.168.59.12 514 #verify connectivity to remote rsyslog server TCP port 50514
 telnet 192.168.59.12 50514 #verify connectivity to remote rsyslog server TCP port 50514

 #On the Centralized rsyslog server
 tail /var/log/messages  
 tail –f /var/log/messages
 netstat –pnltu # Confirm That the Syslog Server Is Listening on Port 514
 ss -tunlp | grep 514
 ss -4altunp | grep 514
 ss -tulnp | grep "rsyslog"
 nc -ulp 514 #cannot telnet to UDP port 514, use netcat 


 #add the necessary UDP and/or TCP firewall rules to allow incoming syslog traffic
 # systemctl restart rsyslog
 # firewall-cmd --add-port=514/udp --permanent
 # firewall-cmd --add-port=514/tcp --permanent
 # firewall-cmd --reload
 ----------------------------------------------------------------------------------------------------
 username must contain fewer than 32 characters and start with a letter
 may consist of letters, numbers, and certain symbols
 ----------------------------------------------------------------------------------------------------
 Deleting a group does not delete all the accounts associated with the group
 Groups may have passwords, these are not account login passwords
 ----------------------------------------------------------------------------------------------------
 /etc/modprobe.d #all the modules and other files, except for the optional configuration files
 stat /lib/modules/`uname -r` #modprobe searches the module directory
 more /proc/modules #a text list of the modules that the system has loaded

 lsmod #list of currently loaded device drivers
 lsmod | wc -l #total loaded Linux kernel modules
 lsmod | grep nvidia #see if Linux kernel drivers (modules) named nvidia loaded or not
 lsmod | egrep -i 'nvidia|e1000e|kvm_intel' #Search for multiple Linux device driver modules

 modinfo e1000 # information about specific driver
 ----------------------------------------------------------------------------------------------------
 #KVM

 $ egrep -c '(vmx|svm)' /proc/cpuinfo | echo "virtualization is  supported" | echo "virtualization is not supported"
 $ egrep -c '(vmx|svm)' /proc/cpuinfo && echo "virtualization is  supported" || echo "virtualization is not supported"
 grep -i vmx /proc/cpuinfo #check if the CPU supports virtualization
 lsmod | grep kvm #check  if the kvm kernel module is loaded

 $ grep -c ^processor /proc/cpuinfo #check that your server has (at least) 8 CPU cores

 To run KVM, you need a processor that supports hardware virtualization. 
 Intel and AMD both have developed extensions for their processors, deemed respectively Intel VT-x (code name Vanderpool) and AMD-V (code name Pacifica)
 #If 0 it means that your CPU doesn't support hardware virtualization.
 #If 1 or more it does - but you still need to make sure that virtualization is enabled in the BIOS. 
 $egrep -c '(vmx|svm)' /proc/cpuinfo  
 $egrep -q 'vmx|svm' /proc/cpuinfo && echo yes || echo no #To use VM drivers, verify that your system has virtualization support enabled
 #If the above command outputs “no”
 #If you are running within a VM, your hypervisor does not allow nested virtualization. You will need to use the None (bare-metal) driver
 #If you are running on a physical machine, ensure that your BIOS has hardware virtualization enabled
 $cat /sys/hypervisor/properties/capabilities #if it is enabled or not from xen

 $kvm-ok #If you see You can still run virtual machines, but it'll be much slower without the KVM extensions
 INFO: Your CPU does not support KVM extensions
 KVM acceleration can NOT be used

 $egrep -c ' lm ' /proc/cpuinfo #If 0 is printed, it means that your CPU is not 64-bit. If 1 or higher it is 64-bit
 $ uname -m
 x86_64

 #By default dhcpd based network bridge configured by libvirtd
 $ brctl show
 $ virsh net-list 

 #All VMs (guest machine) only have network access to other VMs on the same server.
 #A private network 192.168.122.0/24 created
 $ virsh net-dumpxml default

 virt-install --name=linuxconfig-vm \
 --vcpus=1 \
 --memory=2048 \
 --cdrom=/media/sanchez/KARNAK/linux_distributions/CentOS-Stream-8-x86_64-20210617-dvd1.iso \
 --disk size=5 \
 --os-variant=centos-stream8
 ----------------------------------------------------------------------------------------------------

 wc -mlw file1.txt file2.txt #Count words, characters, and lines in multiple files 	
 ls -l *.pdf | wc -l #Count a Certain Type of Files in a Directory
 wc -m yourTextFile # count the total number of characters	
 wc -w yourTextFile #count the number of words
 $ wc -c file1.txt #the number of characters in a file
 $ wc -l file1.txt #the number of lines in a file
 $ head -5 .bash_history | wc -w # the number of words in the first 5 lines of the file

 # filecount=$(ls | wc -l)
 # echo $filecount
 ----------------------------------------------------------------------------------------------------
 # terminal1
 $ ls > pipe2
 $ mkfifo pipe5 -m700
 $ ls -l > pipe5
 $ rm pipe5
 $rm dir1 #remove a non-empty directory

 # terminal2
 $ ls -lart pipe2 # list hidden files in the current directory
 prw-rw-r-- 1 vagrant vagrant 0 Feb 25 15:33 pipe2
 $ ls -lart pipe5
 prwx------ 1 vagrant vagrant 0 Feb 25 20:52 pipe5
 $ cat < pipe5
 total 15828
 prw-rw-r-- 1 vagrant vagrant        0 Feb 25 15:33 pipe2
 prw-rw-r-- 1 vagrant vagrant        0 Feb 25 20:51 pipe4
 prwx------ 1 vagrant vagrant        0 Feb 25 20:52 pipe5
 -rw-rw-r-- 1 vagrant vagrant 16207833 Jan 22 22:02 terraform_0.12.20_linux_amd64.zip
 ----------------------------------------------------------------------------------------------------
 pushd #stores a directory path in the directory stack,adds directory paths onto a directory stack (history), allows you to navigate back to any directory in history
 pushd +2 #use the directory index in the form pushd +# or pushd -# to add directories to the stack and move into
 popd #removes the top directory path from the same stack
 popd +1 #remove a directory from the directory stack inded use popd +# or popd -#
 dirs #display directories in the directory stack (or history)
 dirs -v

 pushd $(pwd) && cd /opt
 popd

 ----------------------------------------------------------------------------------------------------
 #from windows to linux copy problem fix
 $ make
 Makefile:21: *** missing separator.  Stop.
 $ perl -pi -e 's/^  */\t/' Makefile

 # unix/windows file editing
 "/bin/bash^M: bad interpreter: No such file or directory"
 fix:  sed -i -e 's/\r$//' build_all.sh

 ----------------------------------------------------------------------------------------------------
 # ls do not provide creation time but change time
 $ type ll
 ll is aliased to `ls -alF'

 # -a option shows all hidden files and directories (Those who start with .")
 #the -F classify the results in files and folders,makes it more visual when a lot of files and directories with different extensions exist
 ls -alF

 ls -i About-TecMint #inode

 The Bash shell feature that is used for matching or expanding specific types of patterns is called globbing
 $ ls -l ????.txt #files whose names are four characters long
 $ ls -l foot????.doc # files whose names are 8 characters long, first 4 characters are f, o, o and t and extension is doc
 $ ls -l best.??? #all files with the name ‘test’ having any extension of three characters long


 $ ls –lt #lists files in long listing format, and sorts files based on modification time, newest first
 $ ls –lth
 $ ls –ltr #list down files /folders sorted with modified time, -r reverse order
 $ ls -ltr | grep "`date | awk '{print $2" "$3}'`" #todays date
 $ ls -ltr | grep "$(date +"%b %e")"
 $ ls -ltr | grep "Feb 18" #current date "Mar 22" # list files on specific dates
 $ ls -ltr | awk '$6 == "Feb" && $7 >=15 && $7 <= 31 {print $6,$7,$8,$9}' # list files after Feb 15t
 $ ls -ltr . | awk '$6 == "Feb" && $7 >=15 && $7 <= 31 {print $6,$7,$8,$9}' # list files after Feb 15t on specific directory

 $ ls -lrt /var/log | awk '{ total += $5 }; END { print total }' # sum of file sizes

 $ ls -FaGl /var/log/apt | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}') #total size 
 $ ls -FaGl /var/log/apt | sudo tee /dev/stderr | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}') #list directory contents and total size 
 $ ls -laUR /var/log/apt | grep -e "^\-" | tr -s " " | cut -d " " -f5 | awk '{sum+=$1} END {print sum}' #only sum up file sizes not the directory itself

 # total size, sum of files listed
 $ sumcol()
 > {
 >     awk "{sum+=\$$1} END {print sum}"
 > }
 $ ls -lrt /var/log/apt/ | sumcol 5

 $ ls -l | grep 'Mar 22 12:27' | tr -s ' ' | cut -d ' ' -f9 | xargs rm -rf #delete files on specific date

 #Listing of files in directory based on last modification time of file’s status information, or the 'ctime'
 #list that file first whose any status information like: owner, group, permissions, size etc has been recently changed.
 $ ls –lct #List Files Based on Last Modification Time 
 $ ls –ltu #Listing of files in directory based on last access time, i.e. based on time the file was last accessed, not modified.

 #Sorting Ouptut of ls -l based on Date
 #based on 6th field month wise, then based on 7th field which is date, numerically
 ls -l | sort -k6M -k7n 
 ls -l | head -n 10 | sort -k6
 ls -l | head -n 10| sort -k6M -k7n #based on 6th field month wise, then based on 7th field which is date

 ls -lt --time=birth #sorted by creation/birth date time
 ls -l --time=creation #sorted by creation/birth date time

 $ ls -l *.pl #all files of ‘pl’ extension

 $ ls -l [p-s]* #all files and folders whose name contains p or q or r or s
 $ ls -l [1-5]* #all files and folders whose name starts with any digit from 1 to 5

 $ ls -l {?????.sh,*st.txt} #files whose names are 5 characters long and the extension is ‘sh’ or the last two characters of the files are ‘st’ and the extension is ‘txt’
 $ rm {*.doc,*.docx} #delete all files whose extensions are ‘doc’ or ‘docx’

 $ ls a*+(.bash|.sh) #filenames which are starting with character ‘a’ and has the extension ‘bash’ or ‘sh’

 ls -alt #list files in last modifed date order use the -t flag which is for 'time last modified'.
 ls -altr #list files in last modifed date order use the -t flag which is for 'time last modified', reverse order
 ----------------------------------------------------------------------------------------------------
 echo test > >(cat) #the output of echo would be redirected to the file that serves as the input to cat, and cat would produce the contents of that file on standard output

 echo foo | cat -n
 echo foo > >(cat -n) # emulate pipe above

 The process substitution >(command) will be replaced by a file name. 
 This file name corresponds to a file that is connected to the standard input of the "command" inside the substitution

 $ cat .profile | while read line; do ((counter1++)); done
 $ echo $counter1
 $ while read line; do ((count++)); done < <(cat ~/.profile)
 $ echo $count
 101
 ----------------------------------------------------------------------------------------------------
 #Network Troubleshooting 
 Step 1: Check if your interface is configured
 $ ifconfig

 sudo resolvconf -u

 Step 2: Setting up your interface
 check if the drivers are loaded
 $ dmesg | grep -3 -i eth
 configure the interface
 ifconfig eth0 128.42.14.176 netmask 255.255.255.0 up
 Assign a Broadcast to Network Interface
 ifconfig eth0 netmask 255.255.255.224
 If the loopback interface is not up
 ifconfig lo up
 now be able to ping your own machine
 $ ping -c 3 127.0.0.1

 Step 3: Check if you can ping the gateway
 ping the DNS server

 Step 6: Setting up routing
 # route add -net <naddr> netmask <maddr> eth0
 # route add default gw <gaddr> eth0
 setup the loopback route if it's missing
 # route add -host 127.0.0.1 lo

 #Adding and removing routes
 ip ro add 10.0.0.0/16 via 192.0.2.253 
 ip ro del 10.0.0.0/16 via 192.0.2.253
 ip ro
 ip -6 ro #IPv6
 ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}' # IP reaching the public internet, multiple IPs interfaces
 ip ro get $dst_ip from $src_ip #Check routing path
 ip ro get 192.0.0.0

 $ ip route show
 default via 10.0.2.2 dev eth0 proto dhcp metric 100
 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
 192.168.18.0/24 dev eth1 proto kernel scope link src 192.168.18.9 metric 101

 traffic to anywhere else should be forwarded through eth0 to the gateway at 10.0.2.2
 traffic to 10.0.2.2 (the gateway to the public Internet) should be forwarded directly to its destination through eth0
 traffic to anywhere within 192.168.18.0/24 (the local area network) should be forwarded directly to its destination through eth1

 $ ip route get to 192.168.18.12 from 192.168.18.9 iif eth1
 $ ip route get to 192.168.18.9  from 192.168.18.12 iif eth1

 $ route -n
 route add dest_ip_address gateway_address #Add a route to a destination through a gateway
 route add subnet_ip_address/24 gateway_address #Add a route to a /24 subnet through a gateway
 route -t add dest_ip_address/24 gateway_address #Run in test mode (does not do anything, just print)
 route flush #Remove all routes
 route delete dest_ip_address/24 #Delete a specific route

 Step 7: Name resolution
 3 files: /etc/host.conf, /etc/hosts, /etc/resolv.conf

 /etc/host.conf:
 order hosts,bind
 multi on

 /etc/hosts:
 127.0.0.1      localhost loopback
 <IPaddr>        this.host.name

 /etc/resolv.conf:
 domain yourdept.yourcompany.com
 search yourdept.yourcompany.com yourcompany.com
 nameserver <domainaddr>

 ----------------------------------------------------------------------------------------------------
 # ping a host
 ping 192.168.0.2
 #show routing table without resolving domain names
 netstat -nr
 netstat -r -n # The flag U indicates that route is up and G indicates that it is gateway 
 netstat -alun | grep 161
 # show informations about errors/collisions
 netstat -ni  
 # show statistics about your network card
 netstat -i -I em0  
 netstat -a
 netstat -at
 netstat -s
 netstat -au
 netstat -l
 netstat -lu
 netstat -lt
 netstat -tulpn  	
 netstat -plan
 netstat -plan | grep ":80"
 netstat -lntp | grep ':8080.*java' > /dev/null && command
 netstat -pan -A inet,inet6 | grep -v ESTABLISHED #determine which ports are listening for connections from the network
 netstat -tlnw #Use the -l option of the netstat command to display only listening server sockets:
 netstat -plnS #Scan for Open SCTP Ports
 netstat -nl -A inet,inet6 | grep 2500 #Scan for Open SCTP Ports
 netstat -pant | grep -Ei 'apache|:80|:443'
 netstat -tunlp | grep ":80 "
 List all TCP sockets and related PIDs
 netstat -antp
 netstat -anp
 List all UDP sockets and related PIDs
 netstat -anup
 # find out on which port a program is running
 netstat -ap | grep ssh
 #If there is an IP address instead, then the port is open only on that specific interface
 #For listening ports, if the source address is 0.0.0.0, it is listening on all available interfaces
 #The Recv-Q and Send-Q fields show the number of bytes pending acknowledgment in either direction
 #the PID/Program name field shows the process ID and the name of the process responsible for the listening port or connection
 netstat -anptu 
 #number of established connection
 netstat -an|grep ESTABLISHED|awk '{print $5}'|awk -F: '{print $1}'|sort|uniq -c|awk '{ printf("%s\t%s\t",$2,$1); for (i = 0; i < $1; i++) {printf("*")}; print ""}'

 #see that the Nessus server is up and running
 netstat -n | grep tcp
 netstat -tap | grep LISTEN
 netstat -pltn | grep 8834
 ----------------------------------------------------------------------------------------------------
 #ss is the socket statistics command that replaces netstat
 ss -tr #netstat -t
 ss -ntr #see port numbers
 ss -an |grep LISTEN #netstat -an |grep LISTEN
 ss -an | grep 2500 #show SCTP open ports
 ss -tlw # list open ports in the listening state
 ss -plno -A tcp,udp,sctp #The UNCONN state shows the ports in UDP listening mode
 ----------------------------------------------------------------------------------------------------

 # find route to example.com
 traceroute www.example.com
 #find route to example.com using tcptraceroute (which uses tcp to discover path)
 tcpdraceroute www.example.com
 # The maximum number of hops can be adjusted with the -m flag.
 traceroute -m 255 obiwan.scrye.net
 # adjust the size of the packet that is sent to each hop by giving the integer after the hostname
 traceroute google.com 70

 Specify Gateway
 sudo traceroute  -g 10.0.2.2 yahoo.com
 traceroute -g 192.5.146.4 -g 10.3.0.5 35.0.0.0
 #shows the path of a packet that goes from istanbul to sanfrancisco through the hosts cairo and paris
 #The -I option makes traceroute send ICMP ECHO probes to the host sanfrancisco
 #The -i options sets the source address to the IP address configured on the interface qe0
 traceroute -g cairo -g paris -i qe0 -q 1 -I sanfrancisco 
 ip r / ip route #gateway / router
 ip r | grep default #default gateway
 route -n # The flag U indicates that route is up and G indicates that it is gateway
 route -nee
 netstat -r -n # The flag U indicates that route is up and G indicates that it is gateway 
 routel #list routes
 routel | grep default #default gateway


 Specify Source Interface
 sudo traceroute  -i eth0 yahoo.com
 Autonomous Systems
 traceroute  -A  yahoo.com

 tracepath yahoo.com
 tracepath -n yahoo.com
 tracepath -b yahoo.com
 sets the initial packet length
 tracepath -l 28 yahoo.com
 set maximum hops (or maximum TTLs) to max_hops
 tracepath -m 5 yahoo.com
 set the initial destination port to use
 tracepath -p 8081  yahoo.com

 hostname -f #show fully qualified domain name (FQDN)
 hostname #shortname
 hostname -I #the primary/first IP address
 hostname -I | awk '{print $1}' # get primary/first IP IP address of server
 hostname -I | awk '{print $2}' # get second IP IP address of server

 real-time view of the current state of your system
 $ htop

 $ timedatectl
 $ timedatectl list-timezones
 $ sudo timedatectl set-timezone 'Africa/Lubumbashi'

 # Enable NTP synchronization
 timedatectl set-ntp true
 timedatectl status

 # show connected sockets
 sockstat -c 
 # show listening sockets and processes
 sockstat -l

 # show arp table
 arp -a #Show current arp tabl
 arp -na 
 ip neighbour #Show neighbors (ARP table)
 arp -d 192.168.0.2 # delete a record from arp table
 sudo arp -a -d #Clear the entire cach
 arp -s 192.168.0.2  00:10:b5:99:bf:c4 #add a static record in arp table
 # listen on em0 network interface and sniff packets that pass via em0
 $ sudo arp -i eth0


 find out reachability of an IP on the local Ethernet with arping i.e send ARP request 192.168.1.1:
 $ sudo arping -I eth0 -c 3 192.168.18.12
 $ sudo arping -I eth1 -c 3 192.168.18.12
 Find duplicate IP
 $ sudo arping -D -I eth1 -c 3 192.168.18.12

 ping -I eth0 8.8.8.8 #ping 8.8.8.8 using eth0 as a source interface 
 ping google.com
 ping -6 hostname/IPv6 #request IPv6 or IPv4 address
 ping -4 hostname/IPv4
 #The default interval between each ping request is set to one second. You can increase or decrease that time using the –i switch
 #decrease the ping interval, use values lower than 1
 ping -i 0.5 google.com 
 ping -s 1000 google.com #use -s to increase the packet size from the default value of 56 (84) bytes.
 ping -f hostname-IP #use ping flood to test your network performance under heavy load
 ping -c 2 google.com #Limit Number of Ping Packets
 ping -w 25 google.com #stop printing ping results after 25 seconds
 ping -c 10 -q google.com #The letter “q” in this command stands for “quiet” output.
 ping -D google.com #Add Timestamp Before Each Line in ping Output

 $ file *
 20:30:               empty
 file1:               ASCII text
 file2:               ASCII text
 $ file raw_file.txt #if a text file contains ASCII or Unicode
 $ file -b symbolic_test1.txt
 symbolic link to test1.txt
 -----------------------------------------------------------------------------------------------------
 # parse html with curl
 curl -s https://miloserdov.org/ | grep -E -o '<h3 class=ftitle>.*</h3>' | sed 's/<h3 class=ftitle>//' | sed 's/<\/h3>//'
 curl http://test.com | sed -rn 's@(^.*<dd>)(.*)(</dd>)@\2@p'

 cat - > file.html << EOF
 <div class="tracklistInfo">
 <p class="artist">Diplo - Justin Bieber - Skrillex</p>
 <p>Where Are U Now</p>
 </div><div class="tracklistInfo">
 <p class="artist">toto</p>
 <p>tata</p>
 </div>
 EOF
 cat file.html | tr -d '\n'  | sed -e "s/<\/div>/<\/div>\n/g" | sed -n 's/^.*class="artist">\([^<]*\)<\/p> *<p>\([^<]*\)<.*$/artist : \1\ntitle : \2\n/p'
 cat file.html | grep -A2 -E -m 1 '<div class="tracklistInfo">'
 cat file.html | grep -A2 -E -m 1 '<div class="tracklistInfo">' | tail -n1
 cat file.html | grep -A2 -E -m 1 '<div class="tracklistInfo">' |  tail -n2 | head -n1
 cat file.html | grep -A2 -E -m 1 '<div class="tracklistInfo">' |  tail -n2 | head -n1 | sed 's/<[^>]*>//g'
 -----------------------------------------------------------------------------------------------------
 Check what ethernet devices exist currently
 # ls -al /sys/class/net
 # ls -Rl
 list hidden files and the contents of all subdirectories
 ls -aR /home/username
 # ls -pu
 see eth* devices
 # ls -al /sys/class/net
 Get the PCI address of the NIC
 # lspci | grep Mellanox
 # lspci | grep Eth 

 iw distinguishes between wireless LAN hardware devices (the physical layer, referred to as phy) and the network interface configured to use that hardware (e.g. wlan0, 
 similar to an Ethernet eth0 interface). To see the list of devices, and interfaces for each device
 #iw dev

 **MAC_ADDRESS	08:00:27:e3:b0:01
 $ ip link show eth1
 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:e3:b0:01 brd ff:ff:ff:ff:ff:ff
 $ ip addr show eth1
 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:e3:b0:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.253/24 brd 192.168.1.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fee3:b001/64 scope link
       valid_lft forever preferred_lft forever
       
 #temporarily set the IP address
 ifconfig eth0 192.168.8.185
 ifconfig eth0 192.168.8.185 netmask 255.255.255.0 up
 #temporarily change the MAC address
 ifconfig eth0 down hw ether AA:BB:CC:DD:EE:FF && ifconfig eth0 up
 ifconfig  eth0 netmask 255.255.255.0
 ifconfig  eth0 broadcast 192.168.70.255

 ip addr show -> List IP address of the server
 ip addr show eth0
 ip addr | grep inet6 #check that your server supports IPV6
 ip addr show eth1 | grep "inet "
 ip addr add 10.132.1.1/24 dev eth1 ->  Add a new IP4 address
 ip addr show eth1 -> confrm that the new address is available on the interface

 ip link set eth2 down -> bring an interface down  
 ip link set eth2 up

 ip -s link->view basic network statistics on all interfaces
 ip -s link ls eth0 ->see the statistics for the eth0 interface
 ip -s -s link ls eth0 ->see additional info

 ss -t ->show established TCP connections
 ss -u ->show established UDP connections
 ss -A tcp 
 ss -x 
 ss -ltn ->see which ports are listening for connections
 ss -nt
 ss -ltn
 ss -ua
 ss -a -A udp
 ss -lun ->udp
 ss -s->prints out the statistics

 #Install sysstat package
 # /etc/default/sysstat ENABLED="true"
 #sudo service sysstat restar
 vmstat 1 99999 ->the system statistics every second, for the number of times specifed (99999 in this instance)
 vmstat –a 1 99 ->show memory usage information
 vmstat -a -S M 1 9 -> reformat in Mega Bytes
 vmstat 1 99999 ->gather information for disks and other block devices
 vmstat -d -w
 iostat 1 9 ->CPU information and disk information for all devices
 iostat -d -p sda 1 9-> show information for device sda with disk statistics
 sar -u 1 30 -> display CPU statistics every second for 30 seconds 
 sar -r 1 30  -> display memoru statistics every second for 30 seconds 
 sar -b 1 30  -> display block device statistics every second for 30 seconds 

 #=====================================================================

 # curl the binay version info from latest download and later wget the file with
 $curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
 | grep browser_download_url \
 | grep linux-amd64 \
 | cut -d '"' -f 4 \
 | wget -qi -
 $ ls
 prometheus-2.37.0.linux-amd64.tar.gz
 $tar xvf prometheus*.tar.gz
 $cd prometheus*/


 #specify directory and rename the file
 wget --output-document="/home/my_new_file_name" http://someurl
 #add the appropriate BeeGFS repositories
 wget -o /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/beegfs_2015.03/dists/beegfs-rhel7.repo
 wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- |  apt-key add -

 wget example.com/big.file.iso  #start download and stop download ctrl+c key pair
 wget -c example.com/big.file.iso  #resume download 

 wget ‐‐continue example.com/big.file.iso #Resume an interrupted download previously started by wget itself
 wget ‐‐continue ‐‐timestamping wordpress.org/latest.zip #Download a file but only if the version on server is newer than your local copy
 wget ‐‐page-requisites ‐‐span-hosts ‐‐convert-links ‐‐adjust-extension http://example.com/dir/file #Download a web page with all assets - like stylesheets and inline images - that are required to properly display the web page offline.
 wget -q  http://somesite.com/TheFile.jpeg #-q: Turn off wget's output
 wget http://example.com/images/{1..20}.jpg # Download a list of sequentially numbered files from a server
 wget -m -r -linf -k -p -q -E -e robots=off http://127.0.0.1 # Download a complete website
 wget ‐‐mirror ‐‐domains=abc.com,files.abc.com,docs.abc.com ‐‐accept=pdf http://abc.com/ #Download the PDF documents from a website through recursion but stay within specific domains.
 wget ‐‐execute robots=off ‐‐recursive ‐‐no-parent ‐‐continue ‐‐no-clobber http://example.com/ #Download an entire website including all the linked pages and files
 wget ‐‐level=1 ‐‐recursive ‐‐no-parent ‐‐accept mp3,MP3 http://example.com/mp3/ #Download all the MP3 files from a sub-directory
 wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains some-site.com --no-parent www.some-site.com #Download Entire Website
 wget ‐‐recursive ‐‐no-clobber ‐‐no-parent ‐‐exclude-directories /forums,/support http://example.com #Download all files from a website but exclude a few directories
 wget --reject=png www.some-site.com #Reject file types while downloading
 wget -r -A .pdf http://some-site.com/ #Download all PDF files from a website
 wget -r -H --convert-links --level=NUMBER --user-agent=AGENT URL #Download With Wget Recursively,declare a user agent such as Mozilla (wget –user-agent=AGENT)
 wget -e https_proxy=xx.xx.xx.xx:8080 https://example.com/  #use proxy server with wget

 wget -S --spider http://www.uniqlo.com/ #Only Header Information

 ##if link exists
 url="https://www.katacoda.com/courses/kubernetes/launch-single-node-cluster"
 if wget --spider "$url" 2>/dev/null; then #2> /dev/null silences wget's stderr output
  echo "URL exists: $url"
 else
  echo echo "URL does not exist: $url"
 fi

 #connect to a remote server,start download on the remote server,disconnect from the remote server,let it run on the background
 $ nohup wget -q url &  

 wget -i file.txt #Read download URLs from a file,useful in a shell script.

 #one liner if condition
 wget --spider http://192.168.50.15/${distribution}_${codename}_oscap_report.html 2>/dev/null && echo "link exists" || echo "link does not exist"

 wget --spider -S "www.magesh.co.in" 2>&1 | awk '/HTTP\// {print $2}' #see only the HTTP status code
 wget --spider -o wget.log -e robots=off --wait 1 -r -p http://www.mysite.com/ #crawl a website and generate a log file of any broken links

 wget --spider https://example.com/filename.zip 2>&1 | grep Length #file download size without downloading the actual file
 wget ‐‐spider ‐‐server-response http://example.com/file.iso #Find the size of a file without downloading it (look for ContentLength in the response, the size is in bytes)

 wget ‐‐output-document - ‐‐quiet google.com/humans.txt #Download a file and display the content on the screen without saving it locally
 wget ‐‐server-response ‐‐spider http://www.labnol.org/ #the last modified date of a web page (check the LastModified tag in the HTTP header)
 wget ‐‐output-file=logfile.txt ‐‐recursive ‐‐spider http://example.com #Check the links on your website to ensure that they are working. The spider option will not save the pages locally.
 wget ‐‐limit-rate=20k ‐‐wait=60 ‐‐random-wait ‐‐mirror example.com # limited the download bandwidth rate to 20 KB/s and the wget utility will wait anywhere between 30s and 90 seconds before retrieving the next resource.
 wget -O index.html  --certificate=OK.crt --private-key=OK.key https://example.com/ #Client SSL Certificate
 wget -q -O - --header="Content-Type:application/json" --post-file=foo.json http://127.0.0.1 # POST a JSON file and redirect output to stdout
 wget -O wget.zip http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Download file with different name
 wget -o download.log http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #redirect the wget command logs to a log file using ‘-o‘ switch.
 wget ‐‐output-document=filename.html example.com #Download a file but save it locally under a different name
 wget ‐‐directory-prefix=folder/subfolder example.com #Download a file and save it in a specific folder
 wget -r -l inf -A .png,.jpg,.jpeg,.gif -nd https://jekyllrb.com # Download all images of a website
 wget -r --level=1 -H --timeout=1 -nd -N -np --accept=mp3 -e robots=off -i musicblogs.txt #take a text file of your favourite music blogs and download any new MP3 files
 wget --ftp-user=User --ftp-password=Mir URL # FTP download
 wget http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.14.tar.gz.sig #Download multiple file with http and ftp protocol
 wget -i /wget/urls.txt #Read URL’s from a file
 wget -Q10m -i download-list.txt #Setting Download Quota
 wget -c http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Resume download
 wget -b /wget/log.txt http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Download files in background
 wget -b -c --tries=NUMBER URL #number of tries (wget –tries=NUMBER), continue partial download (wget -c)
 wget -b --limit-rate=SPEED -np -N -m -nd --accept=mp3 --wait=SECONDS http://www.uniqlo.com/ #no parent to ensure you only download a sub-directory (wget -np),update only changed files (wget -N), mirror a site (wget -m), ensure no new directories are created (wget -nd), accept only certain extensions (wget –accept=LIST) 
 wget -c --limit-rate=100k  /wget/log.txt http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Limit download speed
 wget --http-user=username --http-password=password http://some-network.net/some-file.txt #Options –http-user=username, –http-password=password
 wget --ftp-user=username --ftp-password=password ftp://some-network.net/some-file.txt #–ftp-user=username, –ftp-password=password
 wget --tries=75 http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Increase Retry Attempts.
 wget ‐‐refer=http://google.com ‐‐user-agent="Mozilla/5.0 Firefox/4.0.1" http://nytimes.com #Wget can be used for downloading content from sites that are behind a login screen or ones that check for the HTTP referer and the User-Agent strings of the bot to prevent screen scraping.
 wget ‐‐cookies=on ‐‐save-cookies cookies.txt ‐‐keep-session-cookies ‐‐post-data 'user=labnol&password=123' http://example.com/login.php_ _wget ‐‐cookies=on ‐‐load-cookies cookies.txt ‐‐keep-session-cookies http://example.com/paywall #Fetch pages that are behind a login page. You need to replace user and password with the actual form fields while the URL should point to the Form Submit (action) page.
 wget ‐‐span-hosts ‐‐level=inf ‐‐recursive dmoz.org #
 wget -r --level=inf -p -k -E --span-hosts --domains=domainA,domainB http://www.domainA #download an entire site (domain A) when its resources are on another domain, (domain B)
 wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA #
 wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA #
 wget --recursive --level=inf --page-requisites --convert-links --adjust-extension --span-hosts --domains=domainA,domainB domainA #

 #=====================================================================
 sudo permission denied
 The redirection to a file is handled by bash. It does therefore not inherit permissions granted by sudo.
 "sudo tee" for writing to a file as root.

 lsblk -f -> when used with the -f option, it prints file system type on partitions 
 sudo file -sL /dev/sdb1 -> file system type on partitions
 lsblk -f
 lsblk -l
 lsblk --scsi
 lsblk -o name,type,fstype,label,partlabel,model,mountpoint,size
 lsblk --json | jq -c '.blockdevices[]|[.name,.size]'

 #/etc/fstab explained, Each field can be separated by another either by spaces or tabs

 First field – The block device,reference a block device is by using its LABEL or UUID (Universal Unique IDentifier)
 $ lsblk -d -fs /dev/sdb1 # get UUID

 Second field – The mountpoint
 Third field – The filesystem type
 Fourth field – Mount options, use the default set of mount options we specify default as a value
 Fifth field – Should the filesystem be dumped?, either 0 or 1,used by the dump backup program (if installed)

 Sixth field – Fsck order;fsck utility, should check filesystems on boot;
 value of 1 must always be used for the root filesystem
 if not  root filesystem,for all the others, value of  2
 If not provided it defaults to 0


 # generate traces of the i/o traffic on block devices 
 "sudo blktrace -d /dev/sda -o - | blkparse -i -"


 #writing ISO usb bootable
 sudo umount /dev/sdX
 sudo dd if=/path/to/ubuntu.iso of=/dev/sdX bs=4M && sync
 sdx-> lslbk command
 sync-> sync bit is important as dd can return before the write operation finishes.

 # mount all file systems on /etc/fstab
 mount -a
 mount -fav
 cat /proc/mounts
 # format linux swap partition
 mkswap
 --------------------------------------------------------------------------------------------------------------------
 #search man pages

 man -k . #List all available man pages on the system and provide a short description
 man -f virt-manager # Keyword must be the exact name of a man page
 man -K date -> display a list of all manual pages containing the keyword "date"
 man -wK word ->list out all manual files with some word.
 man -wK --regex "ipv4.*listen"
 man --regex ".*network$" 

 #search through all man pages on a system for the specified keyword and output the location of any man pages that contain that keyword
 man -S 1 --regex "^virsh" 

 man yum | less +/"install @" #search "install @" pattern in man yum page
 man -P 'less -p install ' yum #search "install" pattern in man yum page
 --------------------------------------------------------------------------------------------------------------------
 # Clone / Compile specific kernel
 sudo git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux -> latest stable kernel to "linux" directory
 git tag -l | grep v4.9.1 -> find specific kernel version
 git checkout -b kernel490 v4.9.1 -> switch to kernel with custom name "kernel490"


 #tail -f vs less +F
 #tail -f only reads the end of the file
 #tail -f doesn't keep the whole file in memory.
 tail -f /var/log/syslog
 #less +F reads the whole file
 #less +F impractical for very large files.
 #less +F can highlight, search, navigate through file
 #hit Ctrl-c to go to “normal” less mode (as if you had opened the file without the +F flag)
 #the search with /foo,next or previous occurrence with n or N, up and down with j and k,, create marks with m
 #hit F to go back to watching mode
 less +F /var/log/syslog

 less -n +F #read only the end of the file

 tail -100f /var/log/messages | grep -V ACPI | grep -i ata #real-time monitoring,tailing 100 lines from the end
 tail -f /var/log/nginx/access.log /var/log/nginx/access.log #watching multiple files

 #top 10 internet protocol addresses hitting a webserver from the access log
 cat /var/log/nginx/access.log | cut -f 1 -d ' ' | sort | uniq -c | sort -hr | head -n 10

 multitail /var/log/auth.log /var/log/kern.log #shows the contents of log files horizontally
 multitail -s 2 /var/log/auth.log /var/log/kern.log #view the contents of log files vertically in two columns
 multitail -s 2 /var/log/syslog, /var/log/kern.log, /var/log/daemon.log and var/log/messages
 multitail -s 2 -ci green /var/log/auth.log -ci blue /var/log/kern.log
 lnav /var/log/syslog /var/log/messages
 lnav a.zip b.zip
 --------------------------------------------------------------------------------------------------------------------
 #crontab
 Display scheduled jobs for the specified user
 crontab -l -u vagrant
 crontab -l
 # Display Cron Table
 ls -la /etc/cron*

 sudo crontab -u user -e #
 crontab -e #when running as a non-root user
 sudo crontab -e #the root user's crontab


 #If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command
 #If the /etc/cron.allow file does not exist but the /etc/cron.deny file does, 
 #then users must not be listed in the /etc/cron.deny file in order to run crontab
 #If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command
 /etc/cron.deny # If a blank cron.deny file has been created, cron only available to root or users in cron.allow. 

 # Delete All Cron Jobs
 crontab -r
 crontab -r -i #the command prompt  to confirm
 # All scripts in each directory are run as root
 #Cron jobs may not run with the environment, in particular the PATH, that you expect. Try using full paths to files and programs
 #The "%" character is used as newline delimiter in cron commands. If you need to pass that character into a script, you need to escape it as "\%".
 #anacron uses the run‑parts command and /etc/cron.hourly, /etc/cron.weekly, and /etc/cron.monthly directories.
 #anacron itself is invoked from the /etc/crontab file
 #user crontabs
 /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/, and /etc/cron.monthly/
 ls -la /etc/cron.daily/ #View daily cron jobs

 crontab -l > backup_cron.txt #Backup All Cron Jobs

 /etc/crontab #not recommended that you add anything,this could cause a problem if the /etc/crontab file is affected by updates
 /etc/cron.d #not be affected by updates, several people might look after a server, then the directory /etc/cron.d is probably the best place to install crontabs
 /etc/cron.d #These files also have username fields

 #the files inside /etc/cron.d
 chown root:root /etc/cron.d/*
 chmod go-wx /etc/cron.d/*
 chmod -x /etc/cron.d/*

 $ cat /etc/cron.allow
 barak
 $ sudo systemctl restart cron
 $ cat /etc/cron.d/barak_job
 */1 * * * * barak echo "Nightly Backup Successful: $(date) " > /tmp/test1_job.log
 */1 * * * * /usr/bin/free -m | awk '{ if($1 == "Mem:" ) print $3}' | awk '{ if ( $1 > 140 ) print $0; else print "less" }' >> /tmp/memo.log

 #troubleshooting cron
 $ cat /etc/cron.d/barak_job
 */1 * * * * barak echo "Nightly Backup Successful: $(date) " &> /tmp/test1_job.log #redirect stdout and stderr to a file.
 */1 * * * * barak echo "Nightly Backup Successful: $(date) " > /tmp/test1_job.log2 2>&1 #redirect stdout and stderr to a file.

 #php specific
 php /bla/bla/something.php >> /var/logs/somelog-for-stdout.log
 #the only difference from the syntax of the user crontabs is that the line specifies the user to run the job as
 00 01 * * * rusty /home/rusty/rusty-list-files.sh #run Rusty's command script as user rusty from his home directory.

 /usr/bin/php /home/username/public_html/cron.php #Execute PHP script:
 mysqldump -u root -pPASSWORD database > /root/db.sql #MySQL dump:
 /usr/bin/wget --spider "http://www.domain.com/cron.php"  #Access URL:

 $ cat /etc/cron.allow
 barak
 $ sudo systemctl restart cron/crond
 $ cat /etc/cron.d/barak_job
 */1 * * * * barak echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log
 $ crontab -u barak -l
 #*/1 * * * * barak echo "Nightly Backup Successful: $(date) runs" >> /tmp/barak_job.log
 $ sudo tail -f /var/log/syslog | grep --color=auto CRON

 crontab -e
 @hourly echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log

 #"-u borg" is used take the identity of the borg user
 # cat /etc/cron.daily/borgbackup_check 
 #!/bin/bash
 sudo -u borg borg check /borgbackup >> /var/log/borgbackup.log

 #once every 5 minutes
 cat | sudo tee /etc/cron.d/cron-mrtg << EOF
 */5 * * * * env LANG=C /usr/bin/mrtg /etc/mrtg.cfg
 EOF
 #verify
 cat /etc/cron.d/cron-mrtg
 crontab -l 

 cat | sudo tee /etc/cron.d/sysinfo << EOF
 #once every 5 minutes
 */5 * * * * /bin/bash /home/vagrant/sysinfo_func.sh
 EOF
 #verify
 cat /etc/cron.d/cron-mrtg

 * * * * * /bin/date >> /tmp/cron_output #This will append the current date to a log file every minute.
 * * * * * /usr/bin/php /var/www/domain.com/backup.php > /dev/null 2>&1 #run a script but keep it running in the background
 at specific time
 00 15 * * 4 sh /root/test.sh
 35 21 * * 7 /bin/date >> /tmp/cron_output
 every 5 minutes
 */5 * * * *  mycommand
 an hourly cron job but run at minute 15 instead (i.e. 00:15, 01:15, 02:15 etc.):
 15 * * * * [command]
 once a day, at 2:30am:
 30 2 * * * [command]
 once a month, on the second day of the month at midnight (i.e. January 2nd 12:00am, February 2nd 12:00am etc.):
 0 0 2 * * [command]
 on Mondays, every hour (i.e. 24 times in one day, but only on Mondays):
 0 * * * 1 [command]
 three times every hour, at minutes 0, 10 and 20:
 0,10,20 * * * * [command]
 # Stop download Mon-Fri, 6am
 0 6 * * 1,2,3,4,5 root          virsh shutdown download
 */5 * * * * /path/to/some-script.sh #every 5 minutes
 @reboot /scripts/script.sh #tasks to execute on system reboot
 @hourly /scripts/script.sh #execute on an hourly.
 0 * * * */scripts/script.sh #execute on an hourly.
 @daily /scripts/script.sh # execute on a daily basis.
 0 2 * * * /scripts/script.sh # executes the task in the second minute of every day.#
 @weekly /bin/script.sh #execute on a weekly basis
 0 0 4 * sun /bin/script.sh #execute on a weekly basis
 @monthly /scripts/script.sh #execute on a monthly basis
 0 0 1 * * /scripts/script.sh #execution of a task in the first minute of the month
 @yearly /scripts/script.sh #schedule tasks on a yearly basis.
 @yearly /scripts/script.sh #executes the task in the fifth minute of every year.
 * * * * *  sleep 15; /scripts/script.sh #schedule a cron to execute after every 15 Seconds
 0 4,17 * * mon,tue /scripts/script.sh #execute twice on Monday and Tuesday
 0 17 * * mon,wed  /script/script.sh #run each Monday and Wednesday at 5 PM
 0 7,17 * * * /scripts/script.sh #execute at 7 AM and 5 PM daily
 0 5 * * mon  /scripts/script.sh #execute the task on every Monday at 5 AM
 0 */6 * * * /scripts/script.sh #run a script for 6 hours interval
 0 8-10 * * * /scripts/script.sh # run every hour between 08-10AM
 0 2 * * sat  [ $(date +%d) -le 06 ] && /script/script.sh #execute on first Saturday of every month
 0   12  1-7 *   *   [ "$(date '+\%a')" = "Mon" ] && echo "It's Monday" #on the first Monday of every month
 * * * feb,jun,sep *  /script/script.sh #run tasks in Feb, June and September months
 --------------------------------------------------------------------------------------------------------------------
 tail -n 1 /usr/share/dict/words #limit the number of lines to show
 tail -c 24 /usr/share/dict/words #limit the number of bytes to show
 tail /usr/share/dict/words /usr/share/dict/french #show the last ten lines of multiple files
 tail -n 5 file_1 file_2
 tail -n +N <filename>   #the lines starting from line number N
 tailf -F /var/log/syslog #monitoring the log file even at its rotation
 tail -q /usr/share/dict/words /usr/share/dict/french #suppress the header line pass the -q option
 ls -t /etc | tail -n 5  #show the five files or folders modified the longest time ago

 #brctl show -> Bridge connections
 --------------------------------------------------------------------------------------------------------------------
 #LVM
 pvdisplay
 pvck
 pvs
 lvscan
 lvdisplay
 lvmdiskscan
 vgchange
 vgscan -a y
 e4defrag -cv /path/to/myfiles (defrag folder )


 $ sudo pvcreate /dev/sdb
 $ sudo pvs
  PV         VG       Fmt  Attr PSize   PFree
  /dev/sda2  centos   lvm2 a--  <63.00g 4.00m
  /dev/sdb   vg_iscsi lvm2 a--  <30.00g    0
 $ sudo pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               vg_iscsi
  PV Size               30.00 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              7679
  Free PE               0
  Allocated PE          7679
  PV UUID               hG93NW-gvRB-njUP-pgj8-omRF-YzFe-rTMWOz

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               <63.00 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              16127
  Free PE               1
  Allocated PE          16126
  PV UUID               rFHI2D-fvZw-Mf2P-gKTC-ZTwt-vdiY-TEQc14
  
 $ sudo vgcreate vg_iscsi /dev/sdb
 $ sudo vgdisplay
  --- Volume group ---
  VG Name               vg_iscsi
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <30.00 GiB
  PE Size               4.00 MiB
  Total PE              7679
  Alloc PE / Size       7679 / <30.00 GiB
  Free  PE / Size       0 / 0
  VG UUID               j63noX-S9I0-5Gp0-3FPg-IZ23-oZNK-6qpb7X
 $ sudo lvcreate -l 100%FREE -n lv_iscsi vg_iscsi
 [vagrant@vg-suricata-30 ~]$ sudo lvscan
  ACTIVE            '/dev/vg_iscsi/lv_iscsi' [<30.00 GiB] inherit
  ACTIVE            '/dev/centos/swap' [2.00 GiB] inherit
  ACTIVE            '/dev/centos/home' [<20.01 GiB] inherit
  ACTIVE            '/dev/centos/root' [40.98 GiB] inherit
 $ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg_iscsi/lv_iscsi
  LV Name                lv_iscsi
  VG Name                vg_iscsi
  LV UUID                exEdIG-s2bK-vFEa-fD3X-dplu-q2W3-1rTXsE
  LV Write Access        read/write
  LV Creation host, time vg-suricata-30, 2019-12-18 12:35:56 +0000
  LV Status              available
  # open                 0
  LV Size                <30.00 GiB
  Current LE             7679
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:3  

 $ sudo vgremove vg_iscsi
 $ sudo pvremove /dev/sdb
 --------------------------------------------------------------------------------------------------------------------
 #troubleshooting prometheus

 #Shutting down Prometheus
 lsof -n -i4TCP:9090 #find a process (Prometheus) listening on port 9090 
 pgrep -f prometheus
 curl -X POST :9090/-/quit
 curl -X POST http://localhost:9090/-/quit #  (when the --web.enable-lifecycle flag is enabled)

 #Prometheus Reload 
 curl -i -XPOST localhost:9090/-/reload 
 killall -HUP prometheus

 ./promtool check config prometheus.yml #check if prometheus.yml is valid

 journalctl --boot | grep prometheus

 #reload Prometheus config file
 kill -HUP 9783
 --------------------------------------------------------------------------------------------------------------------

 #list open files
 lsof
 #list open files owned by user1
 lsof -u user1
 #list open file via tcp
 lsof -i TCP:1-1024
 lsof -i TCP:80
 PID 27808
 lsof -Pan -p 27808 -i
 lsof -p 2


 # troubleshooting #1
 find all the opened files and processes along with the one who opened them
 # lsof –p PID
 Count number of files & processes
 # lsof -p 4271 | wc -l
 Check the currently opened log file
 lsof –p | grep log
 Find out port number used by daemon
 # lsof -i -P |grep 4271

 # find out what running processes are associated with each open port on Linux
 netstat -nlp|grep 9000
 sudo ss -lptn 'sport = :80'
 sudo netstat -nlp | grep :80
 sudo lsof -n -i :80 | grep LISTEN
 fuser 3306/tcp
 fuser 80/tcp
 ss -tanp | grep 6379
 fuser -v -n tcp 22
 sudo netstat -ltnp | grep -w ':80'
 netstat -tulpn | grep :80
 netstat -tulpn
 ls -l /proc/1138/exe 
 sudo ss -tulpn
 sudo ss -tulpn | grep :3306
 fuser 7000/tcp
 ls -l /proc/3813/exe 
 man transmission
 whatis transmission
 # find out current working directory of a process pid 3813
 ls -l /proc/3813/cwd
 pwdx 3813
 # Find Out Owner Of a Process on Linux
 cat /proc/3813/environ
 grep --color -w -a USER /proc/3813/environ
 lsof -i :80 | grep LISTEN
 # The file /etc/services is used to map port numbers and protocols to service names
 grep port /etc/services
 grep 443 /etc/services

 #Start a Linux Process or Command in Background
 $ tar -czf home.tar.gz .
 $ tar -tvf home.tar.gz # list the contents of a .tar file
 $ bg
 $ jobs
 OR
 $ tar -czf home.tar.gz . &
 $ jobs
 #Keep Linux Processes Running After Exiting Terminal
 $ sudo rsync Templates/* /var/www/html/files/ &
 $ jobs
 $ disown  -h  %1
 $ jobs
 OR
 $ nohup tar -czf iso.tar.gz Templates/* &
 $ jobs
 #Detach a Linux Processes From Controlling Terminal
 firefox </dev/null &>/dev/null &

 count & # count command running on the background
 jobs
 fg
 bg
 fg %#    #Replace the # with serial number of the job,bring any job in the foreground 
 fg %2 #bring job 2 into the foreground
 jobs -l
 count 2> /dev/null &

 $ tail -f temp.log  #Placing a Foreground Job into the Background,suspend the job with a Ctrl-Z,
 ^Z[1]+ Stopped tail -f temp.log  
 $ bg # bg to place the suspended job in the background
 $ jobs # list the jobs in the background


 $ jobs -l # list job in the background, process id 105231
 [1]+ 105231 Running 
 $ fg 1 # bring job #1 in the foreground from the background, process id 105231
 sudo rsync 

 $ fg 1 #type ctrl+z to send the job #1 to the background, process id 105231
 sudo rsync 
 ^Z
 [1]+  Stopped 

 $ jobs -l # list the job #1 in the background which is stopped, process id 105231
 [1]+ 105231 Stopped 

 $ bg 1 # run the job 1 in the background again, process id 105231
 [1]+ sudo rsync 

 $ jobs -l # list the job #1 in the background, process id 105231
 [1]+ 105231 Running
 -----------------------------------------------------------------------------------------------------
 2 is the file descriptor of stderr
 the integer file descriptors associated with the streams stdin, stdout, and stderr are 0, 1, and 2, respectively.

 a number 0 = standard out (i.e. STDIN)
 a number 1 = standard out (i.e. STDOUT)
 a number 2 = standard error (i.e. STDERR)
 if a number isn't explicitly given, then number 1 is assumed by the shell (bash)

 ">" send to as a whole completed file, overwriting target if exists

 "echo test > file.txt"  is equivalent to "echo test 1> file.txt"
 echo test 2> file.txt #redirect stderr to file.txt

 "/dev/null" is the null device it takes any input you want and throws it away. 
 It can be used to suppress any output

 "2>/dev/null"
 Redirect STDERR to /dev/null (nothing shows up on console)
 The general form of this one is "M>/dev/null", where "M" is a file descriptor number. 
 This will redirect the file descriptor, "M", to "/dev/null".

 ">&" is the syntax to redirect a stream to another file descriptor

 "echo test 1>&2"   equivalent to "echo test >&2"


 "2>&1"
 The general form of this one is "M>&N", where "M" & "N" are file descriptor numbers. 
 It combines the output of file descriptors "M" and "N" into a single stream.
 "&" indicates that what follows and precedes is a file descriptor, and not a filename.

 "2>&-"
 closing a file descriptor used with redirection
 The general form of this one is "M>&-", where "M" is a file descriptor number. 
 This will close output for whichever file descriptor is referenced, i.e. "M"

 "|&"
 Redirect STDERR and STDOUT to STDIN
 This is just an abbreviation for "2>&1 |"

 "&>/dev/null"
 Redirect both STDERR & STDOUT to /dev/null (nothing shows up on console)
 This is just an abbreviation for >/dev/null 2>&1. 
 It redirects file descriptor 2 (STDERR) and descriptor 1 (STDOUT) to /dev/null

 ">/dev/null"
 Redirect STDOUT to /dev/null (only STDERR shows on console)
 This is just an abbreviation for 1>/dev/null. 
 It redirects file descriptor 1 (STDOUT) to /dev/null.

 "command > /dev/null 2>&1 &"     #Run command in the background, discard stdout and stderr
 "command >> /path/to/log 2>&1 &" #Run command and append stdout and stderr to a log file


 ./command >/dev/null 2>&1 #Hide standard and error outputs

 Hide standard output
 ./command >/dev/null
 sends 2 (stderr) into 1 (stdout), and sends stdout to file.log
 command > file.log 2>&1 
 Hide standard and error outputs and release terminal (run the command in background)
 ./command >/dev/null 2>&1 &
 prevent standard output and error output, redirecting them both to /dev/null
 script > /dev/null 2>&1

 -----------------------------------------------------------------------------------------------------
 #Identify processes using files, directories, or sockets.Who is Using a File or Directory
 $ fuser  .
 $ fuser -v ./
 Check Processes Using TCP/UDP Sockets
 fuser -v -n tcp 5000
 the processes that are using my 'home' directory
 $ fuser ~
 $ fuser ~ -v
 check for the root directory
 $ fuser /
 $ fuser / -v
 $ fuser -v /home/ismail
 $ fuser -v -m /home/ismail/.bashrc
 $ fuser -v -n tcp 8080
 $ fuser -v -n udp 53
 kill this TCP listener, you can use option -k
 $ fuser -i -k 8080/tcp
 shows all processes at the (local) TELNET port
 $ fuser telnet/tcp
 list signals
 $ fuser -l
 STOP a process
 $ fuser -i -k STOP [FILE/DIRECTORY]
 kills all processes accessing the file system /home 
 $ fuser -km /home


 $ who #logged-in users,List Connected Users
 #all the currently logged in users, the time of login and their host machine's IP address.
 #review the contents of the /var/log/utmp file
 $ who -H 
 $ who -r
 run-level 5  2018-07-14 17:16
 $ runlevel ->‘N’ indicates that the runlevel has not been changed since the system was booted. And, "number" is the current runlevel 
 The last run level, and the current run level.
 $ runlevel

 md5sum ubuntu-6.10-desktop-i386.iso
 sha256sum ubuntu-9.10-dvd-i386.iso
 echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check # Validate the kubectl binary against the checksum file
 sha1sum filename #view the SHA-1 of a file
 sha1sum -c DNi70074.bio.sha1 #read SHA1 sums from the FILEs and check
 sha1sum --check --ignore-missing DNi70074.bio.sha1 # Do not fail or report status for missing files

 $ wget https://github.com/coreruleset/coreruleset/archive/refs/tags/v3.3.2.zip
 $ echo -n '88f336ba32a89922cade11a4b8e986f2e46a97cf v3.3.2.zip' | sha1sum -c -
 v3.3.2.zip: OK
 $ sha1sum v3.3.2.zip
 88f336ba32a89922cade11a4b8e986f2e46a97cf  v3.3.2.zip

 #watch is used to run any designated command at regular intervals. 
 watch -n 5 "ls -l | wc l"

 --------------------------------------------------------------------------------------------------------------------
 # detect driver hardware problems
 dmesg | more
 The output of dmesg is maintained in the log file
 /var/log/dmesg
 cat /var/log/dmesg | less
 data from /dev/kmsg
 use syslog
 # dmesg -S
 # limit the output to only error and warnings
 dmesg --level=err,warn
 # dmesg produce timestamps 
 dmesg --level=err -T
 dmesg -T | grep -i eth0
 dmesg --level=err,warn -T | grep -i eth0
 # limit dmesg's output only to userspace messages
 dmesg -u
 # timestmaps along with decode facility and levels in dmesg command output
 dmesg -Tx
 Supported log levels (priorities):
   emerg - system is unusable
   alert - action must be taken immediately
    crit - critical conditions
     err - error conditions
    warn - warning conditions
  notice - normal but significant condition
    info - informational
   debug - debug-level messages
 dmesg -TL -f kern
 dmesg -TL -f daemon
 Supported log facilities:
    kern - kernel messages
    user - random user-level messages
    mail - mail system
  daemon - system daemons
    auth - security/authorization messages
  syslog - messages generated internally by syslogd
     lpr - line printer subsystem
    news - network news subsystem
 # verify vt-d is ON
 "dmesg | grep Virtualization"
 # dmesg | grep -i memory
 # dmesg | grep -i dma
 # dmesg | grep -i usb
 # dmesg | grep -E "memory|dma|usb|tty" 
 # dmesg | grep -E "sda|dma"
 Clear dmesg logs
 # dmesg -C
 # dmesg -c
 Display colored messages
 # dmesg -L
 Monitor real time dmesg logs
 # dmesg --follow
 # dmesg -Tx --follow
 # watch "dmesg | tail 7-20"
 Display raw message buffer
 # dmesg -r
 #virtual machine check
 $ dmesg |grep -i hypervisor
 --------------------------------------------------------------------------------------------------------------------
 $ dmidecode -s system-manufacturer
 --------------------------------------------------------------------------------------------------------------------

 #32x 64x query
 uname –m
 arch
 #linux version
 lsb_release -a
 cat /etc/issue
 cat /etc/os-release
 cat /etc/lsb-release
 cat /etc/*-release
 cat /proc/version
 hostnamectl set-hostname server1
 hostnamectl set-hostname --pretty "Web dev test environment"
 hostnamectl set-hostname --static webdev-test-env
 $ hostname --ip-address
 $ hostname --all-ip-addresses

 dnsdomainname -f



 #create file w multiple lines
 echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  


 sudo ls -hal /tmp/ | sudo tee /tmp/test.out > /dev/null #The redirect to /dev/null is needed to stop tee from outputting to the screen
 sudo ls -hal /tmp/ | sudo bash -c "cat >> /tmp/test.out"  #To append instead of overwriting the output file METHOD1
 sudo ls -hal /tmp/ | sudo tee /tmp/test.out  #To append instead of overwriting the output file METHOD2
 sudo ls -hal /tmp/ | sudo tee --append /tmp/test.out #To append instead of overwriting the output file METHOD2


 # append text with non-root user
 echo "deb http://research.cs.wisc.edu/htcondor/ubuntu/8.8/bionic bionic contrib" |sudo tee -a /etc/apt/sources.list

 #do not see the contents of the website (may receive an access denied or ‘bad bot’ message)
 curl -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36' URL
 curl --compressed URL
 curl URL | iconv -f windows-1251 -t UTF-8 #convert from the windows-1251 encoding to the UTF-8 encoding
 curl -u username:password URL #websites require a username and password to view their content,specify other authentication methods using --ntlm | --digest
 #showing only headers, HTML code not displayed
 curl -s -I -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36' https://www.acrylicwifi.com/AcrylicWifi/UpdateCheckerFree.php?download | grep -i '^location'
 curl -s -v http://www.paterva.com/web7/downloadPaths41.php -d 'fileType=exe&os=Windows' 2>&1 | grep -i 'Location:'

 #save cookies,data transfer using the POST method, the --data option is used
 curl --cookie-jar cookies.txt http://forum.ru-board.com/misc.cgi --data 'action=dologin&inmembername=f123gh4t6&inpassword=111222333&ref=http%3A%2F%2Fforum.ru-board.com%2Fmisc.cgi%3Faction%3Dlogout'
 #get information from a page that only registered users have access to, specify the path to the file with previously saved cookies
 curl -b cookies.txt 'http://forum.ru-board.com/topic.cgi?forum=35&topic=80699&start=3040' | iconv -f windows-1251 -t UTF-8

 #extracting entire archive
 curl -L -o ds.tar.gz https://downloads.dockerslim.com/releases/1.37.3/dist_linux.tar.gz
 tar -xvf ds.tar.gz -C /usr/local/bin

 #uncompress the tar.gz file into a specific folder. 
 tar -xzvf your_file.tar.gz -C /path/to/destination/folder


 tar xvzf tkn_0.10.0_Darwin_x86_64.tar.gz -C /usr/local/bin tkn
 tar xjvf backup.tbz
 tar -zxvf backup.tar.gz
 tar -xf file_name.tar.gz --directory /target/directory

 #ssh and tar to make secure backups. Make a backup via encrypted file transfer
 tar --create --directory /home/joe/tmp/ --file - *| \
 ssh raspberrypi "tar --directory /home/joe \
 --verbose --list --file -"

 wget --no-check-certificate https://www.cacti.net/downloads/cacti-latest.tar.gz
 tar -zxvf cacti-latest.tar.gz
 mv cacti-1* /opt/cacti
 (OR tar -xf cacti-latest.tar.gz --directory /opt/cacti)

 tar -zxvf /tmp/onos-1.12.0.tar.gz  --strip-components 1 --directory /opt --one-top-level=onos
 tar xvf mysql-5.7.23-linux-glibc2.12-x86_64.tar.gz --one-top-level=mysql57 --strip-components 1
 tar zxvf ugly_name.tgz --one-top-level=pretty_name
 #extract .xz file
 unxz tor-browser-linux32-5.5.4_en-US.tar.xz
 tar xvf tor-browser-linux32-5.5.4_en-US.tar
 #extract .bz2 file
 bzip2 -dk FileZilla_3.29.0_x86_64-linux-gnu.tar.bz2 
 tar xvf FileZilla_3.29.0_x86_64-linux-gnu.tar
 #extract .zip file
 unzip terraform_0.11.7_linux_amd64.zip -d terraform
 #extract .rar file
 unrar e extract.rar r
 # create user home directory backup 
 tar cvf filename.tar /home/vagrant/
 # show which files were changed
 tar dvf filename.tar
 # update the changed files
 tar uvf filename.tar 
 # make smaller backup 
 gzip filename.tar


 #format a USB storage device with FAT32 file system
 mkfs –t vfat <USB-device-mount-point>
 # mount -o loop,offset=$((10860003 * 512)) disk.img /mnt
 #find out the USB device mount point
 fdisk -l
 #unmount the drive,you can’t format a mounted drive.
 sudo umount /dev/sdb1
 sudo mkfs.vfat /dev/sdb1
 sudo mkfs.ntfs /dev/sdb1
 mkfs.ext4 <USB-device-mount-point>
 mkfs.ntfs <USB-device-mount-point>
 #Set label name to USB drives
 sudo mkfs.vfat /dev/sdb1 -n sk

 ==========================================================================================================
 #Create a New Sudo User(CentOS)
 adduser username
 passwd username
 usermod -aG wheel username #add user to the wheel group.By default, on CentOS, members of the wheel group have sudo privileges
 su - username # switch to the new user account

 #verify if user is sudoer
 sudo -l -U userjohndoe  #list user's privileges or check a specific command
 sudo --validate / sudo -v #update the user's cached credentials, authenticating the user if necessary
 sudo --list #print the list of allowed and forbidden commands for the user who is executing the sudo command
 groups #verify if user is sudoer, member of wheel group
 sudo whoami # returns root

 ----------------------------------------------------------------------------------------------------
 #Create a New Sudo User (ubuntu)
 sudo adduser barak #create new user
 sudo adduser barak sudo #Add the user to sudo group 
 usermod -aG sudo barak #Add the user to sudo group 

 id barak  #verify sudo group
 groups newuser #verify sudo group

 su - barak #Verify Sudo Access
 $ ls /root
 ls: cannot open directory '/root': Permission denied
 sudo ls /root
 ----------------------------------------------------------------------------------------------------
 echo $HOME $USER
 sudo bash -c 'echo $HOME $USER'
 sudo -H bash -c 'echo $HOME $USER'
 ----------------------------------------------------------------------------------------------------
 cmd:
 sudo sh -c "go build ."

 problem:
 bash: line 1: go: command not found

 fix:
 # indicates that the go command is not in the system's PATH when running the command with sudo. 
 #This is because sudo does not preserve the user's environment variables, including the PATH variable.

 sudo env "PATH=$PATH" sh -c "go build ."
 ----------------------------------------------------------------------------------------------------
 When you use the "su otheruser" command without the hyphen (-), you switch to the specified user's account, 
 but you inherit the current environment variables and working directory of the original user. 
 This can lead to potential issues if you need the new user's environment to be set up completely.

 Example Use Cases:

 If you only need to quickly execute a single command as "otheruser" 
 but do not require the environment to be exactly the same, 
 you might use "su otheruser" to avoid the overhead of starting a new shell session.

 ┌──(vagrant㉿vg-kali-02)-[~]
 └─$ su barak
 Password:
 ┌──(barak㉿vg-kali-02)-[/home/vagrant]
 └─$ echo $HOME
 /home/barak

 ┌──(barak㉿vg-kali-02)-[/home/vagrant]
 └─$ pwd
 /home/vagrant

 ┌──(barak㉿vg-kali-02)-[/home/vagrant]
 └─$ exit
 exit

 ┌──(vagrant㉿vg-kali-02)-[~]
 └─$
 ----------------------------------------------------------------------------------------------------
 When you use the "su - otheruser" command with the hyphen (-), you not only switch to the specified user's account 
 but also start a new shell session with the target user's environment variables and home directory. 
 This ensures a clean environment as if you had logged in directly as that user.

 In general, you should use su - otheruser if you need to change the shell environment. 
 This is especially important if you are going to be running commands that require specific permissions or settings that are not available in the default shell environmen

 Example Use Cases:

 Suppose you are the "originaluser" and you want to run a script that requires specific environment variables 
 and settings of "otheruser." In this case, you would use "su - otheruser" to ensure the script runs with the correct environment.

 ┌──(vagrant㉿vg-kali-02)-[~]
 └─$ su - barak
 Password:
 ┌──(barak㉿vg-kali-02)-[~]
 └─$ echo $HOME
 /home/barak

 ┌──(barak㉿vg-kali-02)-[~]
 └─$ pwd
 /home/barak

 ┌──(barak㉿vg-kali-02)-[~]
 └─$ exit
 logout

 ┌──(vagrant㉿vg-kali-02)-[~]
 └─$
 ----------------------------------------------------------------------------------------------------
 #-H flag makes sudo assume root's home directory as HOME instead of the current user's home directory
 sudo -H 
 #sudo user
 echo "stack ALL=(ALL) NOPASSWD: ALL" |sudo tee -a /etc/sudoers

 #allow a user aaron to run all commands using sudo without a password, open the sudoers file
 $ sudo visudo
 aaron ALL=(ALL) NOPASSWD: ALL 

 %sys ALL=(ALL) NOPASSWD: ALL #all member of the sys group will run all commands using sudo without a password
 alf ALL=(ALL) NOPASSWD: ALL #permit a user to run a given command (/bin/kill) using sudo without a password
 %sys ALL=(ALL) NOPASSWD: /bin/kill, /bin/rm #the sys group to run the commands: /bin/kill, /bin/rm using sudo without a password

 #su vs sudo
 #"sudo" asks for your password,"su" asks for the password for the user whom you are switching to
 #sudo lets you issue commands as another user without changing your identity,entry in /etc/sudoers to execute these restricted permissions
 #without entering the root password
 #su keeps the environment of the old/original user even after the switch to root 
 #creates a new environment (as dictated by the ~/.bashrc of the root user), 
 #similar to the case when you explicitly log in as root user from the log-in screen.
 "su -"  
 "su -l" #pass more arguments

 "su -c" #su [target-user] -c [command-to-run]  a command that you want to run after switching to the target user.
 su -c '/home/annie/annie-script.sh' annie #While logged in as user dave, run the annie-script.sh as user annie
 su -c 'echo I am $(whoami)' #Without specifying a target user,switch into root

 #The password prompt is not preferable, during scripting
 #disable the password prompt when user dave is executing scripts as user annie.dave uses su without having to input annie‘s password.
 #/etc/pam.d/su,add the following lines right after the line "auth sufficient pam_rootok.so" 
 auth  [success=ignore default=1] pam_succeed_if.so user = annie #rule checks if the target user is annie
 auth  sufficient                 pam_succeed_if.so use_uid user = dave #rule to check if the current user is dave
 su -c /home/annie/annie-script.sh annie #run by dave

 auth       sufficient pam_rootok.so
 auth       [success=ignore default=1] pam_succeed_if.so user = otheruser
 auth       sufficient   pam_succeed_if.so use_uid user ingroup somegroup

 #/etc/sudoers
 echo 'dave ALL=(annie) /home/annie/annie-script.sh' | EDITOR='tee -a' visudo #The rule grants dave the permission to execute the script annie-script.sh as user annie on any hosts
 sudo -u annie /home/annie/annie-script.sh #while logged in as dave
 sudo -u root /home/annie/annie-script.sh #Sorry, user dave is not allowed to execute '/home/annie/annie-script.sh' as root
 "sudo -s" or "sudo -i" #mimic "su" or "su -l"
 "sudo -s or sudo -i" #temporarily become a user with root privileges

 #/etc/sudoers
 echo 'dave ALL=(ALL) /home/annie/annie-script.sh' | EDITOR='tee -a' #The rule grants dave to execute the script annie-script.sh as any users
 sudo -u root /home/annie/annie-script.sh #while logged in as dave

 #The password prompt is not preferable, during scripting
 #/etc/sudoers
 dave ALL=(ALL) NOPASSWD: /home/annie/annie-script.sh' | EDITOR='tee -a'

 # switching to root using sudo -i (or sudo su) cancels auditing/logging
 # when a sudo command is executed, the original username and the command are logged
 "sudo su"
 "sudo -i"
 su is equivalent to sudo -i
 gives you the root environment, i.e. your ~/.bashrc is ignored.
 simulates a login into the root account
 Your working directory will be /root
 will read root's .profile


 "sudo -s" 
 gives you the user's environment, so your ~/.bashrc is respected.
 launches a shell as root
 doesn't change your working directory

 "sudo bash" #runs bash as a super user
 sudo -E #The -E (preserve environment) option indicates to the security policy that the user wishes to preserve their existing environment variables. 
 ==========================================================================================================
 mount -l
 lshw -short
 sudo lshw -class disk
 sudo lshw -short -class disk
 lshw -class processor
 file -Ls
 dmesg
 denyhosts
 vmstat
 w
 uptime
 ps
 free
 iostat
 pmap
 paste
 uname
 sudo
 mkdir
 chown
 ptree
 pkill

 killall
 kill -KILL PID
 kill –TERM [PID] #means terminate.the default signal sent by kill,equivalent to kill <PID>

 kill –HUP [PID] #reset or restart the process
 all signals
 $ kill -l
 stop and restart process
 $ kill -1 13980
 1 SIGHUP
 9 SIGKILL stop process without letting gracefully
 15 SIGTERM stop process
 ------------------------------------------------------------------------------------------------------------------
 $ touch mylog
 $ ls -lai mylog
 3145761 -rw-rw-r-- 1 vagrant vagrant 0 Mar 27 21:01 mylog
 update the access time of existing file
 $ touch -c mylog
 Change file access time - 'a' of existing file
 $ touch -a mylog
 Change the modified time '-m' of existing file
 $ touch -m mylog
 $ touch -am mylog
 Set a specific access/modify time instead of current time, specify the datetime in format [[CC]YY]MMDDhhmm[.ss]
 $ touch -c -t 201603051015 mylog
 $ touch -c -d '5 Jan 2009' mylog
 $ touch -c -d '20:30' myfile
 Use the timestamp of another file as reference
 $ touch myfile -r mylog
 use the timestamps of 'apl' for 'apl.c'
 $ touch apl.c -r apl
 $ ls -lai mylog
 3145761 -rw-rw-r-- 1 vagrant vagrant 0 Mar 27 21:06 mylog

 $ stat mylog
  File: 'mylog'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
 Device: fc00h/64512d    Inode: 3145761     Links: 1
 Access: (0664/-rw-rw-r--)  Uid: ( 1000/ vagrant)   Gid: ( 1000/ vagrant)
 Access: 2019-03-27 21:07:54.953000000 +0000
 Modify: 2019-03-27 21:07:54.953000000 +0000
 Change: 2019-03-27 21:07:54.953000000 +0000
 Birth: -
 force touch to not create any new file
 $ touch -c newfile
 $ ls -l newfile
 ls: cannot access 'newfile': No such file or directory
 $ stat newfile
 stat: cannot stat 'newfile': No such file or directory

 ------------------------------------------------------------------------------------------------------------------
 # create symbolic link
 ln -s test1.txt symbolic_test1.txt
 stat symbolic_test1.txt
 touch -c -d '5 Jun 2001' -h symbolic_test1.txt
 stat symbolic_test1.txt
 # detect symbolic link
 touch a.xt
 ln -s a.txt b.txt
 stat b.txt
 file b.txt

 ln -s source.file softlink.file 
 ls -lia #does not share the same inode number and permissions of original file
 rm source.file
 stat softlink.file #No such file or directory

 # create hard link
 ln source.file hardlink.file
 ls -lia #shares the same inodes number and permissions of original file
 rm source.file
 stat hardlink.file
 echo "hola el mundo" >>source.file #change the content of either of the files, the change will be reflected
 cat hardlink.file 
 cat source.file 

 lscpiu
 stat file.txt
 #standard output activities for each available processor
 mpstat
 mpstat -P ALL 2 2
 ----------------------------------------------------------------------------------------------------
 df –h # view the amount of free disk space
 df -HT #Display File System Type
 df -hT /home
 df -t ext3 #display a certain file system type use the ‘-t‘ option
 df -x ext3 #Exclude Certain File System Type
 df -k #usage in 1024-byte blocks, use the option ‘-k‘
 df -m # MB (MegaByte)
 df -i # view number of inodes in the system
 df -kl # get a detail description on disk space usage

 ---------------------------------------------------------------------------------------------------
 du -sh /* # list directory sizes under root / disk
 du -sh /* | sort -h
 du -m /some/path | sort -nr | head -n 20 #sorted list containing the 20 biggest dirs
 for each in $(ls) ; do du -hs "$each" ; done
 du --threshold=1M -h | sort -h #includes hidden dot folders (folders which start with .).
 du -h | sort -h 3

 du -bch #-b gives you the file size instead of disk usage, and -c gives a total at the end
 du -ch | tail -1
 du -sh /some/dir #the summary of a grand total disk usage size of an directory use the option “-s” 
 du -sh /var/* |grep G
 du -ah /home/tecmint # displays the disk usage of all the files and directories
 du -kh /home/tecmint #the disk usage of a directory tree with its subtress in Kilobyte blocks. Use the “-k” (displays size in 1024 bytes units).
 du -kh /home/tecmint #Megabytes (MB)
 du -ch /home/tecmint #The “-c” flag provides a grand total usage disk space at the last line
 du -ah --exclude="*.txt" /home/tecmint
 du -ha --time /home/tecmint #the disk usage based on modification of time, use the flag “–time”

 #The problem with du is that it adds up the size of the directory nodes as well,not to sum up only the file sizes.
 # total size of files in a directory
 du -h -c directory #listing path 
 du -h -c directory|tail -1 #only total size 
 $ du -sh /var/log/apt #only total size and listing 

 #du prints actual disk usage rounded up to a multiple of (usually) 4 KB instead of logical file size
 $ for i in {0..9}; do echo -n $i > $i.txt; done #create files 
 $ ls *.txt
 0.txt  1.txt  2.txt  3.txt  4.txt  5.txt  6.txt  7.txt  8.txt  9.txt
 $ du -ch *.txt | tail -1
 40K     total
 $ ls -FaGl *.txt | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}')
 10

 du /var/* -shc --exclude=lib #--exclude to exclude any directory
 du /var/ -h --exclude=lib --max-depth=1 #first-level sub-directories in the /var/ directory. 
 
 $ du -ch /var/log/apt | tail -1 | cut -f 1
 $ du -ac --bytes /var/log/apt
 $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total, " Bytes" }'
 $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total/1024, " KB" }'
 $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: " total/1024/1024 " MB" }'
 $ du /var/log/apt/*.log | awk '{ print; total += $1 }; END { print "total size: ",total }'
 ----------------------------------------------------------------------------------------------------
 $ dir /var/log/apt #list directory contents
 $ dir /var/log/apt | tee >( awk '{ total += $4 }; END { print total }' ) #list directory contents and total size 
 $ dir /var/log/apt | awk '{ print; total += $4 }; END { print "total size: ",total }' #total size 

 ----------------------------------------------------------------------------------------------------
 #List all running processes containing the string stuff
 cat /proc/cpuinfo
 grep "physical id" /proc/cpuinfo | wc -l
 cat /proc/meminfo
 grep MemTotal /proc/meminfo | awk '{FS=":"}{print $2 }' | awk '{print $1/1024/1024}' 
 cat /proc/zoneinfo
 cat /proc/mounts
 cat /etc/issue

 # missing files
 $ sudo ls -lai /lost+found/
 # errors about a corrupt superblock on the drive
 $ e2fsck -b 8193

 $list files sorted by size
 ls –lSr
 ls -il 
 #get pid of my_app
 my_app & $! 

 pidof lighttpd -> Find PID of A Program/Command
 pidof -s php5-cgi
 pidof -x fetch-data -> Get pids of scripts
 pidof -o 4242 lighttpd ->  ignore or omit processes,useful to ignore calling shell or shell script or specific pid.
 ------------------------------------------------------------------------------------------
 # history,The ! prefix is used to access previous commands.
    !$ - last argument from previous command(last command)
    !:1 # last command's 1st argument
    !:2 # last command's 2nd argument
    !:1-2 # last command's 1st and 2nd argument
    !^ - first argument (after the program/built-in/script) from previous command
    !* - all arguments from previous command
    
    
    !! - previous command (often pronounced "bang bang")
    !n - command number n from history
    !pattern - most recent command matching pattern
    !!:s/find/replace - last command, substitute find with replace
    !3:2 #take the second argument from the third command in the history
    !-5:3 #take the third argument from the fifth last command in the history,Using a minus sign to traverse from the last command of the history

    
 $ echo 'one' 'two'
 one two
 $ !$
 'two'
 $ !^
 'one'
 $ !*
 'one' 'two'
 $ !!
 echo 'one' 'two'
 one two

 $ ls /tmp && cd !*
 ------------------------------------------------------------------------------------------
 $ curl -kL http://localhost/banana
 #curl -IL http://localhost
 HTTP/1.1 200 OK
 Server: nginx/1.10.2

 curl -Is http://www.google.com | head -n 1 #check whether a web site is up, and what status message the web server is showing
 curl -sSf http://example.org > /dev/null
 curl -XGET 'localhost:9200/?pretty'

 curl -X PUT "http://127.0.0.1:9200/mytest_index" #sending data with POST and PUT requests
 curl -d "param1=value1&param2=value2" -X POST http://localhost:3000/data
 curl -d "param1=value1&param2=value2" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://localhost:3000/data
 curl -d "@data.txt" -X POST http://localhost:3000/data
 curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://localhost:3000/data
 curl -d "@data.json" -X POST http://localhost:3000/data

 # check if apache is running
 curl -sf http://webserver/check_url
 # process was holding a particular port open
 ss -tp state listening sport = :80 | grep httpd
 # check a particular process id
 lsof -p 1036 -P | grep 'TCP \*:80'
 $ echo "The process id is" $$
 $ echo "The process id is" $$$$
 # check what process is listening
 $ sudo fuser -n tcp 22
 22/tcp:               1088 14324 14354

 echo $SHELL -> determine current shell type
 cat /proc/cpuinfo | grep 'vmx\|svm' ->  VT-x/AMD-v virtualization is enabled in BIOS 

 # troubleshooting nginx
 journalctl -u nginx.service
 tail -n 50 /var/log/nginx/error.log
 tail -n 50 //var/log/nginx/access.log
 nginx -t
 sudo ss -tulpn # Verify that port 80 or 443 
 curl -I http://10.21.136.13
 curl http://10.21.136.13
 dig +short localhost @8.8.8.8
   ------------------------------------------------------------------------------------------------
   #/etc/systemd/journald.conf
   #SystemMaxUse=100M 
   cat /etc/systemd/journald.conf | grep SystemMaxUse
   journalctl --vacuum-size=100M
   
   sudo usermod -a -G systemd-journal $USER # add the current user to the systemd-journal group
   
   # process the data further with text processing tools like grep, awk, or sed, or redirect the output to a file
   journalctl --no-pager #print its output directly to the standard output instead of using a pager by including the --no-pager flag
   
   journalctl -o json -n 10 --no-pager #change the format to format like JSON
   journalctl -o json-pretty -n 10 --no-pager
   
    cp /etc/systemd/journald.conf{,.orig}
    cp a.txt{,.$(date +%Y%m%d-%H%M)} #clone with date info
    
   sudo mkdir -p /var/log/journal
   ls -l /var/log/journal/3a0d751560f045428773cbf4c1769a5c/
   sudo cp /etc/systemd/journald.conf{,.orig}
   sed -i 's/#Storage.*/Storage=persistent/' /etc/systemd/journald.conf # set "Storage" type to "persistent
   sudo vi /etc/systemd/journald.conf
   Storage=persistent
   systemctl restart systemd-journald.service
   journalctl --flush # move the journal log files from /run/log/journal to /var/log/journal
   #The options prefixed with "Runtime" apply to the journal files when stored on a volatile in-memory file system, 
   #more specifically /run/log/journal
   
   
   journalctl --vacuum-files=2 # have 10 archived journal files and want to reduce these down to 2
   journalctl --verify
   
   journalctl | head -1 #What time range do I have logs for?
   journalctl -F _SYSTEMD_UNIT #What systemd services do I have logs for?  
   journalctl -F _COMM 
   journalctl -F _EXE 
   journalctl -F _CMDLINE
   #What users do the services that logged something run as (swap _UID/-u with _GID/-g for groups)?
   journalctl -F _UID | xargs -n1 id -nu 
   journalctl -F _UID | xargs -n1 id -ng
   
   #provide test input for journalctl
   logger -p err 'something erronous happened'
   systemd-cat -p info echo 'something informational happened'
   
   #What selector fields are there? Show up to 8 values each
   for f in $(sudo journalctl --fields); do 
   echo ===========$f; 
   sudo journalctl -F $f; 
   done | grep -A8 ========
  
   
   #remove all entries
   journalctl --rotate
   journalctl --vacuum-time=1s
   journalctl -m --vacuum-time=1s #-m flag, it merges all journals and then clean them up
   #remove all entries
   find /var/log/journal -name "*.journal" | xargs sudo rm
   systemctl restart systemd-journald
   #remove all entries
   rm -rf /run/log/journal/*

   journalctl --rotate --vacuum-size=500M #rotate journal files and remove archived journal files until the disk space they use is under 500M
   
   
   #Rotating is a way of marking the current active log files as an archive and create a fresh logfile from this moment
   # The flush switch asks the journal daemon to flush any log data stored 
   #in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled.
   #Manual delete,removes all archived journal log files until the last second,clears everything
   journalctl --flush --rotate #applies to only archived log files only, not on active journal files
   journalctl --vacuum-time=1s
   #Manual delete,clears all archived journal log files and retains the last 400MB files
   journalctl --flush --rotate
   journalctl --vacuum-size=400M
   #Manual delete,only the last 2 journal files are kept and everything else is removed
   journalctl --flush --rotate
   journalctl --vacuum-files=2
   
   journalctl -b ->all of the journal entries that have been collected since the most recent reboot
   journalctl --list-boots #list of boot numbers, their IDs, and the timestamps of the first and last message pertaining to the boot
   journalctl --boot=ID _SYSTEMD_UNIT=foo
   journalctl -b -1 -> see the journal from the previous boot,use boot number to pick specific boot	
   journalctl -k -b -1  -> Shows kernel logs for the current boot.
   
   $ journalctl --list-boots #list boot id
   -1 340f8a96d40749f8b2530cc76810d62d Tue 2022-01-18 14:33:46 +03—Tue 2022-01-18 15:14:42 +03
    0 75c35ddeb4274787ad78d1092bf9743a Tue 2022-01-18 23:08:10 +03—Wed 2022-01-19 09:25:35 +03
   $ journalctl -b 75c35ddeb4274787ad78d1092bf9743a #use boot id
   
   journalctl --since "2015-01-10 17:15:00"
   journalctl -S "2020-91-12 07:00:00"
   journalctl -S -1d #The “d” stands for “day”, and the “-1” means one day in the past
   journalctl -S -1h
   journalctl --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00"
   journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00"
   journalctl --since yesterday
   journalctl -S yesterday
   journalctl --since yesterday --until now
   journalctl --since today
   journalctl -S -2d -U today #everything from two days ago up until the start of today
   journalctl --since 09:00 --until "1 hour ago"
   journalctl --since '1h ago' --until '10 min ago'
   
   #syslog log levels i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7)
   journalctl -p 0
   journalctl -p 0..2 # logs for a range between emerg(0) and critical(2)
   journalctl -f -p warning    # show me warnings
   journalctl -p err           # show all errors 
   
   journalctl -xp info
   journalctl -xu sshd
   journalctl -fxu httpd.service
   journalctl -fxu sshd.service -p debug
   journalctl -fx
   journalctl -xn
   
   journalctl /dev/sda ->  displays logs related to the /dev/sda file system.
   journalctl /sbin/sshd #logs from the sshd binary
   journalctl -n20 _EXE=/usr/sbin/sshd
   journalctl /usr/bin/bash
   
   journalctl -u nginx.service -> see all of the logs from an Nginx unit on our system
   journalctl -u nginx.service --since today
   journalctl -b -u docker -o json
   journalctl -u docker.service --since "2016-10-13 22:00"
   journalctl _SYSTEMD_UNIT=sshd.service
   journalctl -u sshd.service
   journalctl -u sshd.service -x #logs with more details
   journalctl _PID=8088
   journalctl -b _SYSTEMD_UNIT=foo _PID=number #logs for systemd-units that match foo and the PID number
   #all messages from the foo service process with the PID plus all messages from the foo1 service
   journalctl -b _SYSTEMD_UNIT=foo _PID=number + _SYSTEMD_UNIT=foo1 
   journalctl -b _SYSTEMD_UNIT=foo _SYSTEMD_UNIT=foo1 #shows logs matching a systemd-unit foo or a systemd-unit foo1
   
   #Filter logs based on user
   id -u www-data 
   33   
   journalctl _UID=33 --since today   
   
   journalctl -k ->Kernel messages, those usually found in dmesg output
   journalctl _TRANSPORT=kernel
   
   journalctl -n 20 ->see with a number after the -n
   journalctl -n 10 -o short-full #Changing the Display Format
   journalctl -n 10 -o verbose
   journalctl -n 10 -o json
   journalctl -n 10 -o json-pretty
   journalctl -n 10 -o cat #see the log entry messages, without time stamps or other metadata
   
   journalctl --disk-usage #using persistent storage then the below output shows the amount of disk used
   #removes archived journal files until the disk space they use falls below the specified size 
   #(specified with the usual "K", "M", "G", "T" suffixes),
   journalctl --vacuum-size=1G
   journalctl --vacuum-time=1weeks #clear all messages older than one week
   journalctl --vacuum-time=2d #Retain only the past two days
   
   journalctl -f -> continuously prints log messages, similar to tail -f  
   journalctl -u mysql.service -f
   journalctl -f -e -p err docker --since today # -e implies -n1000
   
   #who ran sudo in the past week, what commandline, what PWD and user?
   journalctl  --since '1 week ago' _COMM=sudo -o json \
     | jq -r '(.__REALTIME_TIMESTAMP|tonumber|(./1e6)|todate) + "\t" + ._CMDLINE + "\t" + .MESSAGE' \
     | column -ts $'\t'
   #How many ssh auth errors today?   
   journalctl -o cat -p err -u ssh --since today | wc -l
   #filter specific error
   journalctl -o cat -p err | grep "tx hang"
   #executables have been logging errors at a loglevel lower than error in the past month?
   journalctl --since -1month -p 7..4 -o json | jq -r 'select (.MESSAGE | contains("error")) | ._EXE'  | sort -u
   #show error logs for a particular version of a service
   journactl -p err /opt/fooservice/9e76/bin/fooservice
   #Filter by start and end dates and particular PIDs
   journalctl _SYSTEMD_UNIT=docker --since '2018–11–01 14:00' --until '2018–11–13 14:00' _PID=123 _PID=456
   ------------------------------------------------------------------------------------------------
   Job for autofs.service failed because a configured resource limit was exceeded. See "systemctl status autofs.service" and "journalctl -xe" for details.
   systemctl start autofs 
   systemctl is-active autofs
   systemctl is-active autofs >/dev/null 2>&1 && echo YES || echo NO
   
   ps -aux | grep -i autofs | grep -v grep #grep command was shown in the output, remove this distraction is to add another pipe to grep -v grep
   
   kill -9 `ps -ef | grep '[k]eyword' | awk '{print $2}'` # get the pid from ps command
   ps -aux | grep dockerd | grep -v grep | awk '{print $2}'  # get the pid from ps command

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 $ ps aux | grep root #list services running as root
 ps aux | grep 3813
 ps -eo pid,user,group,args,etime,lstart | grep '[3]813'
 ps aux | grep '[1]616'
 ps -eo pid,user,group,args,etime,lstart | grep '[1]616'
 ps aux | grep stuff
 The init process, with process ID 1, which does nothing but wait around for its child processes to die. 
 Usually started for /etc/inittab
 $ ps -ef| grep init
 # see the name of the process
 $ sudo ps 1088 14324 14354
 # CPU time, page faults of child processes
 ps -Sla
 $ ps -lu vagrant
 memory information long format
 $ ps -lma
 signal format
 $ ps -sx
 controlling terminal
 $ ps --tty 1 -s
 #print a process tree
 ps -ejH
 ps axjf

 list of command line arguments
 pstree -a
 show PIDS for each process name
 pstree -p
 sort processes with the same ancestor by PID instead of by name,numeric sort
 pstree -n
 pstree -np
 find out the owner of a process in parenthesis
 pstree -u
 pstree -u vagrant
 pstree -unp vagrant
 highlight the current process and its ancestors
 pstree -h
 highlight the specified process
 pstree -H 60093

 find ID of a process owned by a specific user
 $ pgrep -u vagrant sshd
 $ pgrep -u vagrant -d:
 list process names
 $ pgrep -u vagrant -l
 $ pgrep -u vagrant -a
 count of matching processes
 $ pgrep -c -u vagrant

 top -> Checking the Priority of Running Processes
 ps -o pid,comm,nice -p 594 -> Checking the Priority of Running Processes
 ps -o pid,comm,nice,pri -p $(pidof snmpd)
 ps -fl -C "perl test.pl" -> The “NI” column in the ps command output indicates the current nice value (i.e priority) of a process.
 ps -p 2053 -o comm=

 #NI – is the nice value, which is a user-space concept
 #PRI – is the process’s actual priority, as seen by the Linux kernel
 ps -o ni $(pidof snmpd)

 #Total number of priorities = 140
 #Real time priority range(PR or PRI):  0 to 99 
 #User space priority range: 100 to 139
 ps -o ni,pri $(pidof snmpd)

 #PR = 20 + NI
 #PR = 20 + (-20 to + 19)
 #PR = (20 + -20)  to (20 + 19)
 #PR = 0 to 39  (100 to 139 user space priority range)
 ps -o pid,comm,nice,pri -p $(pidof snmpd)

 cat /proc/$(pidof snmpd)/stat | awk '{print "priority " $18 " nice " $19}'
 ps u $(pgrep snmpd) #ps with headers

 #The NI column shows the scheduling priority or niceness of each process
 #ranges from -20 to 19, with -20 being the most favorable or highest priority for scheduling
 #19 being the least favorable or lowest priority
 ps -e -o uid,pid,ppid,pri,ni,cmd |  { head -5 ; grep snmpd; } #ps with headers

 # priority levels between -20 and 19
 nice -10 perl test.pl -> test.pl is launched with a nice value of 10 when the process is started
 nice --10 perl test.pl -> Launch a Program with High Priority
 nice #Checking default niceness,the default is 0 

 # 0 for none, 1 for real-time, 2 for best-exertion, 3 for inactive
 ionice -c 3 -p 1 #PID as 1 to be an idle I/O process
 ionice -c 2 bash #run ‘bash’  as a best-effort program
 ionice -p 3467 #examine the class and priority used by PID 3467
 ionice -c 1 -n 3 -p 3467 
 #set the I/O scheduling class to Idle,takes longer,no longer performance degradation
 # for pid in $(pidof rsync); do ionice -c 3 -p $pid; done
 ----------------------------------------------------------------------------------------------------- 
 renice -n -19 -p 3534 -> Change the Priority of a Running Process
 #adding more virtual runtime to the process
 #The OS thinks that the process has taken more virtual runtime time than other processes in the run queue.
 #in the next cycle, the CPU gives less time to the process
 #The process finishes late as it’s getting less time “on CPU”
 renice +10 PID
 #The OS thinks that the process hasn’t got enough “on CPU” time than other processes in the run queue
 #in the next cycle, the CPU gives more “on CPU” time to that process as compared to other processes in the run queue.
 renice -10 PID
 /etc/security/limits.conf -> set the default nice value of a particular user or group
 $ pidof rsync
 $ renice +10 2395
 2395 (process ID) old priority 0, new priority 10
 ----------------------------------------------------------------------------------------------------- 
 #Process affinity is the scheduler property that helps to bind or unbind the process so that the process will run only with the allotted CPU
 #Processor affinity, or CPU pinning or “cache affinity”, enables the binding and unbinding of a process or a thread to a central processing unit (CPU) or a range of CPUs

 #According to the taskset command man pages, value f means "any CPU."
 $ pidof rsync
 $ taskset -p 2395 #use the PID to get CPU affinity, returns the current CPU affinity in a hexadecimal bit mask format
 pid 2395's current affinity mask: f
 $ taskset -cp 2395 #get the CPU range of a process
 pid 2395's current affinity list: 0-3

 $ taskset -c 0 vlc #start the VLC program on CPU core ID 0
 taskset 0xa gedit #launch gedit with CPU affinity 0xa.

 #If the server gets a reboot or the process is restarted, the PID changes
 taskset -p 0x11 9030 #assign a process to cores 0 and 4
 taskset -cp 0,4 9030 #assign a process to cores 0 and 4
 taskset -cp 1 9030 # bound the process 9030 to run only on CPU 1, configuration is not permanent
 ----------------------------------------------------------------------------------------------------- 

 gpg --verify gnupg-2.2.3.tar.bz2.sig gnupg-2.2.3.tar.bz2 -> check the signature of the file gnupg-2.2.3.tar.bz2
 ----------------------------------------------------------------------------------------------------- 
 systemd-analyze #the actual boot time of the machine
 systemd-analyze blame #see how long every program and service takes to start up
 systemd-analyze critical-chain # print out the results in a chain of events style
 systemd-analyze critical-chain ntp.service networking.service
 systemd-analyze plot > boot_analysis.svg
 xviewer boot_analysis.svg  
 systemd-analyze time -H [email protected] #view information from a remote host over ssh
 systemd-analyze blame -H [email protected]

 systemd-cgtop #top control groups by their resource usage such as tasks, CPU, Memory, Input, and Output
 ----------------------------------------------------------------------------------------------------- 
 PARSING JSON FILE

 sudo apt-get install -y jq
 curl -s 'https://api.github.com/users/lambda' | jq -r '.name'
 
 grep -w \"key_name\" /vagrant/test.json |tail -1 | cut -d\" -f4
 grep -w \"author\" /vagrant/test.json |tail -1 | cut -d\" -f4

 $ FOOBAZ="tester"
 $ jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}' > test1.json
 $ cat test1.json

 export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"') #fill environment variables from JSON object keys (e.g. $FOO from jq query ".foo")
 echo '{ "foo": 123, "bar": 456 }' | jq '.foo' #print out the foo property
 apod_url=$(curl -s https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY | jq -r '.hdurl') #get the URL of the current Astronomy Picture of the Day (APOD)
 echo '{ "Version Number": "1.2.3" }' | jq '."Version Number"' #if a property has a spaces or weird characters
 echo '[1,2,3]' | jq '.[]' #how iteration works
 echo '[ {"id": 1}, {"id": 2} ]' | jq '.[].id' #access a property on each item
 echo '{ "a": 1, "b": 2 }' | jq '.[]' #the value of each key/value pair
 ----------------------------------------------------------------------------------------------------
 bootstrap.sh

 parted /dev/sdb mklabel msdos
 parted /dev/sdb mkpart primary 512 100%
 mkfs.xfs /dev/sdb1
 mkdir /mnt/disk

 mount /mnt/disk
 # Format the /dev/sdb partition with XFS filesystem and with a GPT partition table
 sudo parted -s /dev/sdb mklabel gpt mkpart primary xfs
 sudo mkfs.xfs /dev/sdb -f
 sudo blkid -o value -s TYPE /dev/sdb
 # list disk UUIDs
 ls -l /dev/disk/by-id
 $ fdisk -v
 $ sudo fdisk -l
 $ sudo fdisk -l /dev/sda1
 -----------------------------------------------------------------------------------------------------
 cut -c3    -> print the character from each 3rd line as a new line of output.
 cut -c2,7 -> Display the 2nd and 7th character from each line of text
 cut -c-4  -> Display the first four characters from each line of text
 cut -c13-  -> Print the characters from thirteenth position to the end.
 cut -d' ' -f4  -> Given a sentence, identify and display its fourth word. Assume that the space (' ') is the only delimiter between words.
 cut -d' ' -f1-3  -> Given a sentence, identify and display its first three words. Assume that the space (' ') is the only delimiter between words.

 cut -f 1-3  -> Given a tab delimited file with several columns (tsv format) print the first three fields.
 cut  -f2-  -> Given a tab delimited file with several columns (tsv format) print the fields from second fields to last field.
 -----------------------------------------------------------------------------------------------------
 uniq -ci -> count the number of times each line repeats itself (only consider consecutive repetions).compare consecutive lines in a case insensitive manner
 uniq -u  -> display only those lines which are not followed or preceded by identical replications

 Given a text file, count the number of times each line repeats itself (only consider consecutive repetions). 
 Display the count and the line, separated by a space. 
 uniq -ci | cut -c7-
 ----------------------------------------------------------------------------------------------------- 
 head -v -n 3 file1 # list the names of the files before outputting their content to the terminal
 head -n2 -q file1 file2 #Use the -n option to print the first n lines from a file
 head -n 20 -> Display the first lines of an input file.
 head -n-2 example.txt # skips the last 2 lines and prints the remaining lines.
 head -n10 filename | tail -5 #prints the lines between numbers 5 and 10
 head -c5 example.txt #prints the first 5 bytes from the file.
 head -c-7 example.txt #skip printing last 7 bytes.
 head -c20 -> Display the first characters of an input file.
 head -n 22 | tail -n +12 -> Display the lines (from line number 12 to 22, both inclusive) of a given text file
 # print the lines between 5 and 10, both inclusive
 cat filename | head | tail -6
 -----------------------------------------------------------------------------------------------------
 tail -n 20 | tail -n +12 -> Display the last lines of an input file.
 tail -c 20 -> Display the last characters of an input file
 -----------------------------------------------------------------------------------------------------
 echo BigcapsSmallCaps | tr [:lower:] [:upper:] # convert string into lower case, capital case etc
 tr '()' '[]' -> In a given fragment of text, replace all parentheses with box brackets 
 tr -d [:lower:] -> In a given fragment of text, delete all the lowercase characters
 tr -d "[:space:]" < raw_file.txt #remove all whitespace characters from the file
 echo -e "   \t  A   \tB\tC   \t  " | tr -d "[:blank:]" #deletes any space or tabulation character
 tr -s  ' ' -> In a given fragment of text, replace all sequences of multiple spaces with just one space
 distribution=$(lsb_release --id | cut -f2 | tr [:upper:] [:lower:]) #all big caps to small caps
 -----------------------------------------------------------------------------------------------------
 sort -> Given a text file, order the lines in lexicographical order.
 sort -r -> Given a text file, order the lines in reverse lexicographical order
 sort -n -> the lines reordered in numerically ascending order
 sort -nr  -> The text file, with lines re-ordered in descending order (numerically). 

 given a file of text,in TSV (tab-separated) format.Rearrange the rows of the table in descending order of the values
 sort -t$'\t' -rnk2
 given a file of tab separated weather data (TSV). There is no header column in this data file.Sort the data in ascending order
 sort -nk2 -t$'\t'
 given a file of pipe-delimited weather data (TSV). There is no header column in this data file.
 sort -nrk2 -t$'|'

 -----------------------------------------------------------------------------------------------------
 paste -s ->  Given a CSV file where each row contains the name of a city and its state separated by a comma.replace the newlines in the file with tabs
 paste - - - ->  given a CSV file where each row contains the name of a city and its state separated by a comma, restructure the file in such a way, that three consecutive rows are folded into one, and separated by tab.
 paste -s -d ";"  ->  given a CSV file where each row contains the name of a city and its state separated by a comma.replace the newlines in the file with semicolon
 paste - - - -d ";" ->  given a CSV file where each row contains the name of a city and its state separated by a comma. restructure the file so that three consecutive rows are folded into one line and are separated by semicolons
 -----------------------------------------------------------------------------------------------------
 #Detect exploitation attempts of the vulnerability in uncompressed files in the Linux logs directory /var/log and all its subdirectories
 egrep -I -i -r '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+' /var/log
 find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -I -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'" \;
 find /var/log/ -name '*.gz' -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'" \;
 #searches for exploitation attempts in compressed files in folder /var/log and all sub folders
 find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+'

 #files starting at the current directory (.) and that up to a maximum of 1 level of subdirectories
 find . -maxdepth 2 -type f -name file.txt | xargs -I{} cat {} > ./total_file.txt

 #Get total size of a list of files
 perl -le 'map { $sum += -s } @ARGV; print $sum' -- *.pdf #Size of all non-hidden PDF files in current directory.

 #list files between 1st Dec 2021 and 1st Jan 2022 and total size of each file
 find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sh {} \;
 find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec ls -lt {} \; | sort -k6M -k7n #sorting month & date based
 find . -name 'flibble*' -ctime +90 -exec du -sh {} \;

 find . -type f -mmin -5 -print0 | xargs -0 /bin/ls -ltr #which files was modified in last 5 minutes
 find . -type f -mmin -5 -exec ls -ltr {} +
 find . -mmin -5 -exec ls -ltrd {} + #not limiting to files

 #list files between 1st Dec 2021 and 1st Jan 2022 and grand total size of each found files, not sum of total sizes
 "find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} +"
 find . "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} + | tail -1 #only total
 find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} + | tail -1 | awk '{print $1}'

 $ find . -size +2G #search for all files greater than 2 Gigabytes
 $ find . -size -10k #search for all files with less than 10 Kilobytes
 $ find . -size +10M -size -20M #search for files greater than 10MB but smaller than 20MB
 $ sudo find /var -size +5M -exec ls -sh {} + #search for files in /etc directory which are greater than 5MB and print file size
 $ find . -type f -exec ls -s {} + | sort -n -r | head -3 #Find first 3 largest files located in a in a current directory recursively
 $ find /etc/ -type f -exec ls -s {} + | sort -n | head -3 #Find first 3 smallest files located in a in a current directory recursively
 $ find . -type f -size 0b # search for empty files
 $ find . -type f -empty # search for empty files

 $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; # find files larger than 1M,`M'    for Megabytes (units of 1048576 bytes)
 $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; | wc -l #get count
 $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; | awk '{ total += $5 }; END { print total }' # get total size, column 5(size) of ls command, 

 #total size of all found files
 $ sudo find /var/log -name \*.log -size +1M -exec ls -l {} \; | awk '{ sum += $5} END  \
 {hum[1024^3]="Gb"; hum[1024^2]="Mb"; hum[1024]="Kb"; for (x=1024^3; x>=1024; x/=1024) { if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x]; break; } } if (sum<1024) print "1kb"; }'


 $ find /var/log/apt -type f -name "*.dat" -size +100M #list files larger than 100M

 $ find /var/log/apt -iname *.log -print0 | xargs -r0 du -csh | tail -n 1; # -iname case insensitive
 $ find /var/log/apt -iname *.log -exec ls -lh {} \;

 $ find /var/log/apt -name *.log -size +10c -print0 | du -c --files0-from=- | awk 'END{print $1}'
 $ find /var/log/apt -name *.log -size +10c -print0 | du -ch --files0-from=- | awk 'END{print $1}'
 $ find /var/log/apt -name *.log -size +10c -print0 | du -ch --files0-from=- --total -s|tail -1 #xargs pipe "|" calls du command many times
 $ find /var/log/apt -name *.log -type f -exec ls -s \; | awk '{sum+=$1;} END {print sum/1000;}' #excludes all directories
 du -ch /var/log/apt | tail -1 | cut -f 1

 $ (find /var/log/apt -name *.log -size +10c -printf '%s+'; echo 0 ) | bc
 $ ( find /var/log/apt -name *.log -size +10c -printf 's+=%s\n'; echo s ) | bc

 $ find /var/log/apt -name *.log -size +10c -printf '%s\n' | jq -s add
 $ find /var/log/apt -name *.log -size +10c -exec stat -c%s '{}' + | jq -s add

 find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -print0 | xargs -0 du -c --block-size=human-readable
 find . -name 'flibble*' -ctime +90 -print0 > filenames && du -shc --files0-from=filenames
 du -c `find . -name 'flibble*' -ctime +90` | tail -1
 find . -name 'flibble*' -ctime +90 -printf "%s\n" |perl -lnE '$sum += $_} END {say $sum'
 find . -name 'flibble*' -ctime +90 -printf "%s\t%p\n" |perl -apE '$sum += $F[0]} END {say $sum'
 echo "$(( ($(find . -name 'flibble*' -ctime +90 -type f -printf '%k+' )0)/1024/1024 )) GB"

 #-mtime +7 means older than 8 days (age rounded to integer number of days greater than 7). 
 log_history=13 && find /opt/freeswitch/var/log/freeswitch -type f -mtime +$log_history -delete #Delete old/rotated log files
 # if tomcat directory exists,delete logs
 log_history=13 && [[ -d /var/log/tomcat7 ]] && find /var/log/tomcat7 -type f -mtime +$log_history -delete 
 #Delete FreeSWITCH wav/opus recordings older than 13 days
 history=13 && find /var/freeswitch/meetings/ -name "*.wav"  -mtime +$history -delete

 # find all files, SUID bit enabled
 find / -perm -4000 -exec ls -l {} \;
 find /usr/bin/ -perm -4000 -exec ls -l {} \;
 find /bin/ -perm -4000 -exec ls -l {} \;
 find / -xdev -perm -4000 2>/dev/null

 #-perm denotes that we will search for the permissions that follow:
 #-u=s denotes that we will look for files which are owned by the root user
 #-type states the type of file we are looking for
 #f denotes a regular file, excluding directories and special files
 find / -perm -u=s -type f 2>/dev/null


 find / -uid 0 –perm -4000 -print #find all programs whose SetUID is set to run as root
 find / -perm -2000 -exec ls -l {} \; # find all files, SGID bit enabled

 find /lib/modules/`uname -r` -type f -name '*quota_v*.ko*'

 #counts files recursively in all subfolders in the specified folder
 find /data -type f | wc -l
 #counts files in the current dir, not recursively
 find /data -maxdepth 1 -type f | wc -l
 #counts folders recursively in all subfolders in the specified folder
 find /data -type d | wc -l

 find -type f -exec md5sum -t {} \; | cut -d ' ' -f 1 | sort | md5sum #compute checksum

 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 #between 1st Dec 2021 and 1st Jan 2022
 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -ls #list files between 1st Dec 2021 and 1st Jan 2022
 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -exec echo {} \; #test before delete
 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -exec rm -rf {} \; #delete between 1st Dec 2021 and 1st Jan 2022
 find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec ls -l {} \;

 #never put the -delete action at the first position
 #If the -delete action is at the first position, during its evaluation, it deletes the given directory and everything in it
 #the -delete action implies the -depth option
 #The -depth option asks the find command to search each directory’s contents before the directory itself. 
 # -delete as the first option, it starts deletion from each directory tree’s very bottom
 $ find test -delete -type d -name '.git' # the test directory has been deleted
 $ ls test
 ls: cannot access 'test': No such file or directory

 #the -delete action cannot delete a non-empty directory recursively, can only delete files and empty directories

 find test -depth -type d -name '.git' -exec rm -r '{}' \; #remove all .git directories 
 find test -type d -name '.git' | xargs rm -r #remove all .git directories 
 find ~/Downloads/ -empty -type d -delete #delete all empty directories
 find /path/ -empty -type d | wc -l ## count empty dirs only ##
 find /path/to/dir/ -type d -empty -print0 | xargs -0 -I {} /bin/rmdir "{}" #find and delete all empty directories
 find /path/to/dir -type d -empty -print0 -exec rmdir -v "{}" \; #find and delete all empty directories,slow due to -exec
 $ sudo find /var -type d -empty -mtime +50
 $ sudo find /var -type d -empty -mtime +5 -exec sh -c 'du -sch' sh {} +

 #-exec with an external command, it fills each found file in the ‘{}’ placeholder
 find test -name 'whatever.txt' -exec rm {} \; #remove all whatever.txt files
 find test -name 'whatever.txt' | xargs rm #remove all whatever.txt files
 find ~/Downloads/ -empty -type -f -delete #delete all empty files
 find /path/ -empty -type f | wc -l ## count empty files only ##

 find /path/to/dir/ -type f -empty -print0 | xargs -0 -I {} /bin/rm "{}" #delete all empty files
 find /path/to/dir/ -type f -empty -print0 -exec rm -v "{}" \; #delete all empty files,slow due to -exec

 find / -name .DS_Store -delete #-delete will perform better because it doesn't have to spawn an external process for each and every matched file
 find / -name ".DS_Store" -exec rm {} \; #recommended because -delete does not exist in all versions of find
 find / -iname "*~"  -exec rm -i {} \; # gives an interactive delete
 find / -name .DS_Store -exec rm {} + #The command termination + instead of \; highly optimizes the exec clause by not running the rm command for each and every .DS_Store present on the file system
 find / -name .DS_Store -print0 | xargs -0 rm #avoiding the overhead of spawning an external process for each matched file

 find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sh {} \;#list files between 1st Dec 2021 and 1st Jan 2022 and total size

 #-delete does not delete empty directories
 $ find /path/to/dir/ -type d -name ".TemporaryItems" -delete
 find: cannot delete ‘./.TemporaryItems’: Directory not empty
 $ find /path/to/dir/ -type d -name ".TemporaryItems" -exec rm -rv "{}" +

 find /home -group ftpusers # list file owned by a user or group
 find /data/project -group ftpusers -name "*.c" # list file owned by all *.c file belongs to a group called “ftpusers
 find /data/project -group ftpusers -name "*.c" # list file owned by all *.c file belongs to a group called “ftpusers ,case insensitive
 find $HOME -name "*.mp4" -group pedro -ls #list file in ls command format pass the -ls
 find /var -user pedro
 find /var/www -user pedro -name "*.pl" # find all *.pl (perl files) file belongs to a user
 find / -type f -user bonnie -o -user clyde #find files by users bonnie and clyde
 find / -type d -user vivek -o -user wendy #find dirs  by users bonnie and clyde

 ----------------------------------------------------------------------------------------------------
 #delete multiple folders in a folder which have files in them

 Run the following command to list the folders that will be affected by the deletion command
 The output of the find command will display a list of folder names. Carefully review this list to make sure that it includes only the folders you intend to delete
 find . -type d

 Once you're satisfied with the list and have made any necessary modifications to exclude specific folders, you can run the original deletion command:
 find . -type d -exec rm -r {} \;


 Use the xargs command to pass the output of the find command to the rm command:
 find . -type d | xargs rm -rf

 find . -type d -exec rm -r {} \;
 find .: Searches for files and directories in the current directory and its subdirectories.
 -type d: Specifies that the search should only include directories (folders).
 -exec rm -r {} \;: Executes the rm -r command on each found directory. The {} is a placeholder for the found directory name, and \; indicates the end of the -exec command.
 Be cautious when using this command, as it will delete all directories, including their contents, within the specified directory.

 find . -type d -exec rm -rf {} \;
 use the -f option with rm to suppress confirmation prompts and force deletion. However, be cautious with the -f option, as it will delete without asking for confirmation.
 ----------------------------------------------------------------------------------------------------


 find test -type d -name '.git' # list git directories
 find . -type d -newermt 2012-02-01 ! -newermt 2022-01-01 -ls #list directories between 1st Dec 2021 and 1st Jan 2022
 find . -type d -newermt 2012-03-22 ! -newermt 2022-03-24 -exec echo {} \; #test before delete
 find . -type d -newermt 2012-02-01 ! -newermt 2022-01-01 -exec rm -rf {} \; #delete directories between 1st Dec 2021 and 1st Jan 2022

 find /dir/ -type f -newerXY 'yyyy-mm-dd'
 The letters X and Y can be any of the following letters:
    a – The access time of the file reference
    B – The birth time of the file reference
    c – The inode status change time of reference
    m – The modification time of the file reference
    t – reference is interpreted directly as a time

 find . -type f -newerat 2017-09-25 ! -newerat 2017-09-26 #all files accessed on the 25/Sep/2017
 find /home/you -iname "*.c" -atime 30 -type f  #all *.c file accessed exactly 30 days ago
 find /home/you -iname "*.c" -atime -30 -type f #all *.c file accessed 30 days ago, not older than 30 days
 find /home/you -iname "*.c" -atime -30 -type f -ls
 find /home/you -iname "*.c" -atime +30 -type f #all *.c file accessed more than 30 days ago, older than 30 days
 find /home/you -iname "*.c" -atime +30 -type f -ls
 -----------------------------------------------------------------------------------------------------
 #Users of the bash shell need to use an explicit path in order to run the external time command and 
 #not the shell builtin variant.  On systemwhere time is installed in /usr/bin,
       
 #Real is wall clock time - time from start to finish of the call. This is all elapsed time including 
 #time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete).

 #User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. 
 #This is only actual CPU time used in executing the process. Other processes and time the process 
 #spends blocked do not count towards this figure.

 #Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time 
 #spent in system calls within the kernel, as opposed to library code, which is still running in user-space

 $ /usr/bin/time -o out.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} \;
 $ cat out.txt
 0.01user 0.03system 0:00.06elapsed 76%CPU (0avgtext+0avgdata 8756maxresident)k
 0inputs+0outputs (0major+2771minor)pagefaults 0swaps

 #the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end
 $ /usr/bin/time -o outplus.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} +;
 $ cat outplus.txt
 0.00user 0.01system 0:00.02elapsed 73%CPU (0avgtext+0avgdata 8804maxresident)k
 0inputs+0outputs (0major+1091minor)pagefaults 0swaps

 $ /usr/bin/time -o out.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} \;
 $ /usr/bin/time -f "\t%C [Command details],\t%K [Total memory usage],\t%k [Number of signals process received]" ping -c 2 howtoforge.com
 -----------------------------------------------------------------------------------------------------
 # displays without comments                                                    
 egrep -v "^#|^$" /etc/zabbix/zabbix_server.conf
 -----------------------------------------------------------------------------------------------------
 #r = recursive i.e, search subdirectories within the current directory
 #n = to print the line numbers to stdout
 #i = case insensitive search
 grep -rni "string" * 
 grep -rni "apache /etc/cron.d"

 #string search current and subfolders
 $ grep -rl "900990" .
 ./.crs-setup.conf.swp
 ./crs/crs-setup.conf
 ./crs-setup.conf

 # displays the comments
 grep ^# /etc/resolv.conf

 # displays without comments                                                    
 grep ^[^#] /etc/resolv.confprint directory/file structure in the form of a tree
 grep ^[^\;] /etc/resolv.conffind . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
 grep -v "^#" /etc/zabbix/zabbix_server.conf | grep -v "^$"sed '' quote.txt -> display the contents of the file

 # search multiple strings, words
 grep 'string1' filename | grep 'string2' #search two strings in one line 
 grep -n 'string1' filename | grep 'string2' #search two strings in one line and print line numbers
 grep 'string1.*string2\|string2.*string1' filename #search two strings in one line 
 grep -n 'string1.*string2\|string2.*string1' filename #search two strings in one line and print line numbers
 grep -E "string1(?.*)string2" file #search two strings in one line 
 grep -nE "string1(?.*)string2" file #search two strings in one line and print line numbers

 #Grep for Multiple Strings
 grep 'wordA*'\''wordB' *.py ### Search all python files for 'wordA' or 'wordB'
 grep 'word*' *.txt ### Search all text files
 grep 'word1\|word2\|word3' /path/to/file
 grep 'warning\|error\|critical' /var/log/messages
 grep -e 'warning\|error\|critical' /var/log/messages
 egrep -wi --color 'warning|error|critical' /var/log/messages #-i (ignore case)
 egrep -wi --color 'foo|bar' /etc/*.conf
 egrep -Rwi --color 'foo|bar' /etc/ #including sub-directories
 egrep -w 'warning|error|critical' /var/log/messages
 grep -w 'warning\|error\|critical' /var/log/messages

 egrep -ne 'null|three' #search multiple string and output line numbers 

 grep -o "0x[^']*" file.txt # matching text starting with "0x"
 grep "zip$" #filters the lines that end in zip

 grep -r --include "*.jar" JndiLookup.class / #Detect the presence of Log4j 

 grep --color regex filename #Highlight
 grep --color ksh /etc/shells
 grep -o regex filename #Only The Matches, Not The Lines
 egrep "v{2}" filename #Match a character “v” two times
 egrep 'co{1,2}l' filename #match both “col” and “cool” words
 egrep 'c{3,}' filename #match any row of at least three letters ‘c’
 grep "[[:digit:]]\{2\}[ -]\?[[:digit:]]\{10\}" filename #match mobile number format 91-1234567890
 egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' file #match an IP address,All three dots need to be escaped

 $ grep '^[P-R]' list.txt #lines from list.txt file that starts with P or Q or R
 $ grep '[^A-C]' list.txt #lines from list.txt file that starts with A or B or C

 $ grep [!P-R] list.txt #from list.txt file that starts with ‘P’ or Q or R
 $ grep [!4-8] list.txt #lines from list.txt file that starts with any digit from 4 to 8.

 $ grep a$ list.txt #lines from list.txt file that ends with ‘a’
 $ grep 50$ list.txt #lines from list.txt file that end with the number 50

 grep -i "boar" /etc/passwd #Perform a case-insensitive search for the word ‘bar’

 grep "Gnome Display Manager" /etc/passwd #If the search string includes spaces, enclose it in single or double quotation marks

 #the string “linux” will match only if it occurs at the very beginning of a line
 grep '^linux' file.txt #The ^ (caret) symbol 
 grep 'linux$' file.txt #lines end with linux string
 grep '^linux$' file.txt #lines contain only linux string
 grep '^\.[0-9]' filename #lines starting with a dot and digit
 grep '^..$' filename #lines with  two characters

 #The . (period) symbol is a meta-character that matches any single character
 grep 'kan..roo' file.txt #match anything that begins with “kan” then has two characters and ends with the string “roo”
 grep 'acce[np]t' file.txt #find the lines that contain “accept” or “accent”
 grep 'co[^l]a' file.txt #match any combination of strings starting with “co” followed by any letter except “l” followed by “la”, such as “coca”, “cobalt” and so on
 grep '^[A-Z]' file.txt #matches each line that starts with a capital letter

 grep 's*right' #match “right”, “sright” “ssright” and so on
 grep -E '^[A-Z].*[.,]$' file.txt #matches all lines that starts with capital letter and ends with either period or comma

 grep 'b\?right' file.txt #match both “bright” and “right”. The ? character is escaped with a backslash because we’re using basic regular expressions
 grep -E 'b?right' file.txt

 grep -E 's+right' file.txt #match “sright” and “ssright”, but not “right”
 grep -E '[[:digit:]]{3,9}' file.txt #matches all integers that have between 3 and 9 digits

 grep 'word' filename
 grep 'word' file1 file2 file3
 grep -i "boar" /etc/passwd #Perform a case-insensitive search for the word ‘bar’

 "grep -R 'httpd' ." #Look for all files in the current directory and in all of its subdirectories
 # search for a keyword in text files within a folder and its subfolders. 
 grep -r "keyword" /path/to/directory
 grep -r "keyword" --include "*.txt" /path/to/directory # search for a keyword in a specific type of file
 grep -r -i "keyword" /path/to/directory #case-sensitive when searching for keywords

 grep -r "192.168.1.5" /etc/ #search recursively i.e. read all files under each directory for a string “192.168.1.5”
 grep -c 'nixcraft' frontpage.md #display the total number of times that the string ‘nixcraft’ appears in a file named frontpage.md

 #Grep NOT
 #-v flag to print inverts the match; that is, it matches only those lines that do not contain the given word
 grep -v -c -e "that" ->  find out how many lines that does not match the pattern
 grep -v Sales employee.txt #all the lines except those that contains the keyword “Sales”

 grep -w "the" -> Output only those lines that contain the word 'the'.
 grep -iw "the" -> Output only those lines that contain the word 'the'. The search should NOT be case sensitive.
 grep -viwe "that" -> Only display those lines that do NOT contain the word 'that'.
 grep -Eiw "th(e|ose|en|at)" < /dev/stdin -> display all those lines which contain any of the following words "the,that,then,those" .The search should not be sensitive to case. Display only those lines of an input file, which contain the required words.  
 grep '\([0-9]\) *\1' -> Given an input file, with N credit card numbers,grep out and output only those credit card numbers which have two or more consecutive occurences of the same digit (which may be separated by a space, if they are in different segments). Assume that the credit card numbers will have 4 space separated segments with 4 digits each

 #top 10 IP addresses in the log file.
 grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" access.log | uniq -ci | sort -nr | head -n10

 ifconfig -a | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | awk 'ORS=NR%2?" , ":"\n"'
 ip addr show eth1 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'
 ifconfig -a | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}"

 #list process binary path and permissions
 ps aux | awk '{print $11}' | xargs -r ls -la 2>/dev/null |awk '!x[$0]++'
 ps -elf | grep autofs | grep -v grep  | awk '{print $4}' | xargs kill -9 

 ######group expressions,grep -E option is for extended regexp,three expressions are functionally equivalent
 grep "\(grouping\)" file.txt #use parentheses without using extended regular expressions, escape with the backslash
 grep -E "(grouping)" file.txt
 egrep "(grouping)" file.txt

 grep -E "(GPL|General Public License)" GPL-3 #find either GPL or General Public License in the text
 grep -E "(copy)?right" GPL-3 #matches copyright and right by putting copy in an optional group
 grep -E '(fear)?less' file.txt #matches both “fearless” and “less”. The ? quantifier makes the (fear) group optional
 grep -E "free[^[:space:]]+" GPL-3 # matches the string free plus one or more characters that are not white space characters
 grep -E "[AEIOUaeiou]{3}" GPL-3 #find all of the lines in the GPL-3 file that contain triple-vowels
 grep -E "[[:alpha:]]{16,20}" GPL-3 #match any words that have between 16 and 20 characters

 grep -e pattern1 -e pattern2 filename #Grep OR Using grep -e,
 egrep 'Tech|Sales' employee.txt #Grep OR Using egrep
 grep 'Tech\|Sales' employee.txt #Grep OR Using \|,grep either Tech or Sales from the employee.txt file
 grep 'fatal\|error\|critical' /var/log/nginx/error.log
 grep -E 'fatal|error|critical' /var/log/nginx/error.log # use the extended regular expression, then the operator | should not be escaped
 grep -E 'Tech|Sales' employee.txt #Grep OR Using -E

 #Grep AND
 grep Manager employee.txt | grep Sales #all the lines that contain both “Manager” and “Sales” in the same line
 grep -E 'Dev.*Tech' employee.txt #all the lines that contain both “Dev” and “Tech” in it (in the same order).
 grep -E 'Manager.*Sales|Sales.*Manager' employee.txt #all the lines that contain both “Manager” and “Sales” in it (in any order)
 -----------------------------------------------------------------------------------------------------
 user@host: $ cat<<EOF > file.txt
 $ > 1 line
 $ > other line
 $ > n line
 $ > EOF
 user@host:
 -----------------------------------------------------------------------------------------------------
 # append text
 cat<<EOF | sudo tee -a ceph.conf
 public network = 192.168.18.0/24
 osd pool default size = 2
 EOF
 -----------------------------------------------------------------------------------------------------
 sudo install consul /usr/bin/consul
 (
 cat <<-EOF
  [Unit]
  Description=consul agent
  Requires=network-online.target
 	After=network-online.target
  [Service]
  Restart=on-failure
  ExecStart=/usr/bin/consul agent -dev
  ExecReload=/bin/kill -HUP $MAINPID
  [Install]
  WantedBy=multi-user.target
 EOF
 ) | sudo tee /etc/systemd/system/consul.service
 -----------------------------------------------------------------------------------------------------
 cat <<EOT | sudo tee /lib/systemd/system/procenv.service
 [Unit]
 Description=Display systemd environment

 [Service]
 Type=oneshot
 ExecStart=/usr/bin/procenv --file=/tmp/procenv-systemd.log
 EOT
 ----------------------------------------------------------------------------------------------------- 
 > outputs to a file
 >> appends to a file
 < reads input 
 <<Here tells the shell that you are going to enter a multiline string until the "tag" Here. You can name this tag as you want, it's often EOF or STOP.
 "EOF" is known as a "Here Tag"

 The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, 
 known as a "here-document", to the input of a command.

 The format of here-documents is:
          <<[-]word
                  here-document
          delimiter
 	  
 If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. 
 This allows here-documents within shell scripts to be indented in a natural fashion.
 -----------------------------------------------------------------------------------------------------
 # Assign multi-line string to a shell variable
 # The $sql variable now holds the new-line characters
 # verify with echo -e "$sql"
 sql=$(cat <<EOF
 SELECT foo, bar FROM db
 WHERE foo='baz'
 EOF
 )
 -----------------------------------------------------------------------------------------------------
 #Pass multi-line string to a file in Bash
 $ cat <<EOF > print.sh
 #!/bin/bash
 echo \$PWD
 echo $PWD
 EOF
 -----------------------------------------------------------------------------------------------------
 # Pass multi-line string to a pipe in Bash
 $ cat <<EOF | grep 'b' | tee b.txt
 foo
 bar
 baz
 EOF
 -----------------------------------------------------------------------------------------------------
 $ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
 # my config file
 foo=bar
 EOF
 -----------------------------------------------------------------------------------------------------
 echo -e "
 Home Directory: $HOME \n
 hello world 1 \n
 hello world 2 \n
 line n... \n
 " > file.txt 
 -----------------------------------------------------------------------------------------------------
 echo write something to file.txt | cat > file.txt
 cat >file.txt <<< Write something here

 # see the line numbers
 cat -n song.txt
 #  shows at the end of line and also in space showing ‘$‘ if there is any gap between paragraphs
 # useful to squeeze multiple lines in a single line.
 cat -e test
 # all output will be redirected in a newly created file
 cat test test1 test2 > test3
 #  Sorting Contents of Multiple Files in a Single File
 cat test test1 test2 test3 | sort > test4
 # Display Multiple Files at Once
 cat test; cat test1; cat test2

 # append your text to the end of the file
 cat >> ~/.bashrc <<EOF
 # my config file
 foo=bar
 EOF

 cat > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor <<EOF
 performance
 EOF
 -----------------------------------------------------------------------------------------------------
 curl -sSL https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_linux_amd64.zip -o nomad.zip
 	  https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip	
 curl -sSL https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_linux_amd64.zip -o vagrant.zip
 curl -sSL https://releases.hashicorp.com/vagrant/2.2.2/vagrant_2.2.2_linux_amd64.zip -o vagrant.zip
 unzip vagrant.zip

 $ curl -L https://raw.githubusercontent.com/do-community/ansible-playbooks/master/docker/ubuntu1804.yml -o /vagrant/docker_ubuntu.yml

 export VER="4.4.6"
 "curl -SL https://github.com/NagiosEnterprises/nagioscore/releases/download/nagios-$VER/nagios-$VER.tar.gz | tar -xzf -"
 cd nagios-$VER
 ------------------------------------------------------------------------------------------
 tar -C /usr/local -xzf go1.19.linux-amd64.tar.gz # extract the archive into /usr/local, creating a fresh Go tree in /usr/local/go
 ------------------------------------------------------------------------------------------
 split --bytes=2048m WinXP.img WinXP_img_
 # four files (2GB each) appeared
 WinXP_img_aa
 WinXP_img_ab
 WinXP_img_ac
 WinXP_img_ad
 cat WinXP_img_* > WinXP.img

 # join smaller files into a larger one
 cat partfilename* > outputfilename

 video.avi.01
 video.avi.02
 video.avi.03
 cat video.avi.* > video1.avi

 $ cat file1
 1. Asia:
 2. Africa:
 3. Europe:
 4. North America:

 tac tacexample.txt #print files in reverse
 $ tac file1
 4. North America:
 3. Europe:
 2. Africa:
 1. Asia:

 $ cat file2
 1. India
 2. Nigeria
 3. The Netherlands
 4. The US
 $ join file1 file2
 1. Asia: India
 2. Africa: Nigeria
 3. Europe: The Netherlands
 4. North America: The US

 join -1 1 -2 1 -e 'empty' /tmp/in /tmp/out | tr " " "\t" #join the two files,the first column in each file as index

 # create a new file “new.txt” that is a concatenation of “file1.txt” and “file2.txt”
 cat file1.txt file2.txt > new.txt
 
 format text and convert it to a different width
 $ fmt --width=20 test.txt
 ------------------------------------------------------------------------------------------
 g++ -v
 g++ temp.cpp
 # run
 ./a.out
 ------------------------------------------------------------------------------------------
 # Installing software from source
 tar xvzf package.tar.gz
 tar xvjf package.tar.bz2
 cd package
 ./configure
 make
 make install
 # Cleaning up
 make clean
 make uninstall

 # "make clean" runs as expected even if you do have a file named clean.
 #  There are two reasons to use a phony target: to avoid a conflict with a file of the same name, and to improve performance. 
 .PHONY: clean
 clean:
 rm -rf *.o

 # case1
 $ cat make
 hello : hello.o
        gcc -Wall hello.o -o hello
 $ cat hello.c
 #include<stdio.h>

 int main(void)
 {
            printf("\n Hello World!!!\n");
                return 0;
 }
 $ cp make mon-makefile
 $ ls
 hello.c  make  mon-makefile

 make -C makefile-test1/ hello
 make -f mon-makefile
 make -s

 # case2
 $ cat file2.h
 void add(int a, int b, void (*f)(int));
 $ cat file2.c
 #include<stdio.h>
 #include"file2.h"

 void add(int a, int b, void(*f)(int))
 {
            int c  = a+b;
                f(c);
 }
 $ cat file1.c
 #include<stdio.h>
 #include"file2.h"

 void callback (int result)
 {
            printf("\n Result is : [%d] \n", result);
 }

 int main(void)
 {
            int a=0,b=0;
                printf("\n Enter two numbers to add: ");
                    scanf("%d %d",&a, &b);
                        add(a,b,callback);
                            return 0;
 }
 $ cat makefile
 file : file1.o file2.o
                gcc -Wall file2.o file1.o -o file

 file1.o : file1.c file2.h
                gcc -c -Wall file1.c -o file1.o

 file2.o : file2.c file2.h
                gcc -c -Wall file2.c -o file2.o
 $ cp make mon-makefile
 $ ls
 file1.c  file2.c  file2.h  makefile  mon_makefile


 ------------------------------------------------------------------------------------------
 # linux system management
 top
 sar
 vmstat
 iostat
 free
 ps
 tcpdump
 iptraf
 nestat

 # /Proc file system - Various Kernel Statistics
 cat /proc/cpuinfo
 cat /proc/meminfo
 cat /proc/zoneinfo
 cat /proc/mounts
 ------------------------- -----------------------------------------------------------------
 $ uptime
 #"system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads.
 #show three averages, for 1, 5, and 15 minutes
 #If the averages are 0.0, system is idle
 #If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing
 #If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.

 ps -eL h -o state | egrep "R|D" | wc -l #The instantaneous number of such tasks
 #Linux load average,the instantaneous load of a system the number of tasks (processes and threads) that are willing to run at a given time t
 #either in state R or D, either actually running or blocked on some resource (CPU, IO, ...) waiting for an opportunity to run
 ------------------------------------------------------------------------------------------
 list user vagrant's full command line of processes
 $ top -c -u vagrant
 ignore idle processes
 $ top -i -u vagrant
 updated with 5 secs intervals, including child processes
 $ top -u vagrant -c -d 5 -S
 #determine which Plaso processes are running
 top -p `ps -ef | grep log2timeline.py | grep python | awk '{ print $2 }' | tr '\n' ',' | sed 's/,$//'`

 #interactive top command
 #If you have a multi-core CPU, press "1" to change the display and see individual statistics for each CPU
 #press “t” to swap the CPU displays to simple ASCII graphs that show the percentage of usage for each CPU
 #Pressing “c” toggles the COMMAND column between displaying the process name and the full command line
 #Press “u” to see the processes for a single user. You’ll be prompted for the name or UID
 #Press I to see only active tasks.
 #press “r” to change the nice value (priority) for a process
 #Press “k” to kill a process. You’ll then be prompted for the process ID you want to kill
 ------------------------------------------------------------------------------------------
 The load averages shown by these tools is read /proc/loadavg file
 cat /proc/loadavg
 
 ------------------------------------------------------------------------------------------
 # same inodes
 ls -ldi /. /..
 2 drwxr-xr-x 24 root root 4096 Feb 21 20:28 /.
 2 drwxr-xr-x 24 root root 4096 Feb 21 20:28 /..

 # different inodes
 ls -ldi /home/vagrant/. /home/vagrant/..
 3145730 drwxr-xr-x 7 vagrant vagrant 4096 Feb 22 10:14 /home/vagrant/.
 3145729 drwxr-xr-x 3 root    root    4096 Aug 24 08:48 /home/vagrant/..
 ------------------------------------------------------------------------------------------
 #ChatGPT

 Both .bashrc and .profile files are used for setting environment variables and defining user-specific configurations on Linux, but they have some differences in their purpose and usage.

 Here are some of the differences between .bashrc and .profile files:

    File Location: .bashrc is a Bash shell-specific initialization file that is located in the user's home directory. On the other hand, .profile is a generic shell initialization file that is also located in the user's home directory.

    Shell Usage: .bashrc is loaded for Bash shell instances only, while .profile is loaded for every shell instance (including Bash, Korn shell, Bourne shell, etc.) that is started.

    Invocation: .bashrc is invoked for non-login shells, while .profile is invoked for login shells.

    Content: .bashrc is typically used for setting Bash shell-specific configurations such as aliases, environment variables, and command prompt settings, while .profile is used for setting environment variables and other configurations that should be applied to all shells.

    Loading order: When a Bash shell instance is started, it first loads the /etc/profile file, which is a system-wide shell initialization file. After that, it loads the user's ~/.bash_profile or ~/.bash_login or ~/.profile file (in that order) if it exists. If none of these files exist, it loads the ~/.bashrc file.

 In summary, .bashrc is used for Bash shell-specific configurations, while .profile is used for general shell configurations that should apply to all shells. .bashrc is loaded for non-login shells, while .profile is loaded for login shells.

 ------------------------------------------------------------------------------------------
 # The user file-creation mode mask (umask)
 /etc/profile 
 ~/.bashrc 
 ~/.bash_profile

 # By default
 0022 (022)
 0002 (002)
 files
 666
 directories
 777

 The default umask 002 used for normal user
 directory permissions
 775
 file permissions
 664

 The default umask for the root user is 022
 directory permissions
 755
 file permissions
 644

 base permissions
 directory permissions
 (rwxrwxrwx) 0777
 file permissions
 (rw-rw-rw)  0666 

 # No other user can read or write your data
 umask 077
 # when you share data with other users in the same group
 # Members of your group can create and modify data files
 # those outside your group can read data file, but cannot modify it.
 umask 022
 # exclude users who are not group members
 umask 007

 # The octal umasks are calculated via the bitwise AND of the unary complement of the argument using bitwise NOT
                Octal value : Permission
                0 : read, write and execute
                1 : read and write
                2 : read and execute
                3 : read only
                4 : write and execute
                5 : write only
                6 : execute only
                7 : no permissions
 ------------------------------------------------------------------------------------------
 # testing webpages

 $ telnet control01 80 #test website availability and get the response code

 GET /index.html HTTP/1.1
 Host: control01

 GET /index.html HTTP/1.1
 Host: control01
 If-modified-since: Sun, 24 Feb 2019 12:59:37 GMT

 GET /telnet-send-get-head-http-request HTTP/1.1
 HOST: control01

 HEAD / HTTP/1.1
 Host: control01
 ------------------------------------------------------------------------------------------
 # delete files containing special chars

 $ cat>>"-f"<<EOF
 > test
 > EOF

 $ ls
 -f  test  test2  videos

 $ ls -li
 total 260
 3145770 -rw-rw-r-- 1 vagrant vagrant      5 Mar  5 17:02 -f

 find / -name wget 2>/dev/null
 find /home/vagrant -name file1
 find /home/vagrant -user root
 find /home/vagrant -group root
 find . -inum 3145770 -delete
 find . -inum 3145770 -exec rm -i {} \;
 ls -il {file-name}
 find . -inum [inode] -exec rm -i {} \;
 ------------------------------------------------------------------------------------------
 # steganography, attaching a .rar file to a .jpg etc.
 cat pic.jpg file.rar > result.jpg
 ------------------------------------------------------------------------------------------
 # view used IRQs 
 cat /proc/interrupts
 #determine the IRQ number associated with the Ethernet driver
 $ grep enp0s3 /proc/interrupts
 19:      24006   IO-APIC-fasteoi   enp0s3
 $ grep eth0 /proc/interrupts
 19:      13247   IO-APIC  19-fasteoi   eth0
 
 # troubleshoot network card etc. hardware conflicts, what addresses are used or free, or move conflicting hardware to free resource
 # listing IRQs currently being used.not listed IRQs are considered free.used when devices alert CPU to take action
 cat /proc/interrupts
 # listing used DMA channel.when devicess access memory directly without going through CPU
 cat /proc/dma
 # listing I/O ports currently being used.any range not listed is free and can be used by other devices.
 # devices have unique I/O addresses
 cat /proc/ioports
  
 ------------------------------------------------------------------------------------------
 # view default shell for each user
 $ cat /etc/passwd
 # There are seven fields in the /etc/passwd file
 # username, UID, GID, comment, home directory, command
 # add an asterisk at the beginning of the password field in the /etc/passwd file, that user will not be able to log in
 /etc/passwd

 /etc/adduser.conf #Set DHOME variable for new default home directory of users.
 # useradd tecmint
 # useradd -m tecmint #create user with default home dir
 # passwd tecmint
 create admin user
 sudo useradd vagrant -s /bin/bash -g sudo -m
 modify a existing user to admin user
 sudo usermod -aG sudo vagrant # add user to a group, add vagrant to sudo group
 create a user ‘anusha‘ with a home directory ‘/data/projects‘.
 # useradd -d /data/projects anusha
 create a user ‘navin‘ with custom userid ‘999‘.
 # useradd -u 999 navin
 add a user ‘tarunika‘ with a specific UID and GID
 # useradd -u 1000 -g 500 tarunika
 add a user ‘tecmint‘ into multiple groups like admins, webadmin and developer
 # useradd -G admins,webadmin,developers tecmint
 # id tarunika
 uid=995(tarunika) gid=1001(vboxadd) groups=1001(vboxadd)
 create a user ‘aparna‘ with account expiry date i.e. 27th April 2014 in YYYY-MM-DD 
 useradd -e 2014-03-27 aparna
 useradd -r -d /opt/gvm -c "GVM User" -s /bin/bash gvm #Create a system account,subordinate uid/gid feature is disabled,mail directory is not created

 chage -l aparna #verify the age of account
 chage -d 0 user #Minimum number of days between password change  
 make  changes to "/etc/shadow" file as root user #Minimum number of days between password change 
 chage -M 7 aparna # aparna account’s password must be changed at least once every seven days

 sudo useradd test-user-0 && sudo passwd -d test-user-0 #create user without password
 awk -F: '($2 == "") {print}' /etc/shadow # verify users without password
 passwd -l test-user-0 #Lock all empty password account
 cat /etc/passwd | cut -d : -f 1 | awk '{ system("sudo passwd -S " $0) }' #list locked users
 sudo passwd -S -a | grep " L " | cut -d " " -f1 #list locked users, ubuntu
 sudo passwd -S -a | cut -d " " -f1-2 | grep "L$" #list locked users, ubuntu
 passwd -l user1 #lock the password
 grep user1 /etc/shadow  #two exclamation mark (!!) before the encrypted password which means that the password has been locked
 passwd -u user1 #unlock the password
 passwd -S user1 #see the lock status of the user

 #A user whose account has a "non-interactive shell" (/bin/false, /sbin/nologin) can't log in interactively 
 #prevents SSH command execution if the user has SSH keys on the system
 #users can't get a shell prompt to run commands
 # users may be able to log in to do read/send email (via POP/IMAP & SMTP AUTH)
 #/etc/security/access.conf
 #service accounts often don't have a "proper" login shell, /usr/sbin/nologin as login shell (or  /bin/false)
 #service accounts have user IDs in the low range, e.g. < 1000 or so. Except for UID 0
 #service accounts are typically locked
 #not possible to login (for traditional /etc/passwd and /etc/shadow)
 #achieved by setting the password hash to arbitrary values such as * or x)

 useradd testuser --shell=/sbin/nologin
 usermod testuser -s /sbin/nologin 
 usermod -s /sbin/nologin test4 #Disable User Account With nologin or false Shells
 usermod testuser -s /bin/false #exit as soon as the user logs in, and return exit status 0 (false),do not receive any kind of message like they would with /sbin/nologin
 adduser subversion --shell=/bin/false --no-create-home
 # directed to the /sbin/nologin shell and receive the following message This account is currently not available.
 usermod testuser -s /bin/bash #enable shell by setting a shell


 #list non-interactive shell users, service accounts, aka technical accounts
 $ awk -F: '{if( $7~"/sbin/nologin" || $7~"/bin/false") print "service account..:",$1,"shell..:" $7}' /etc/passwd

 #The users that have nologin defined as their default shell often have higher privileges,unable to log in directly 
 #limit the damage that a breach of system could suffer
 /sbin/nologin #Limiting system's attack surface
 /bin/false


 set a account password expiry date i.e. 45 days on a user ‘tecmint’
 # useradd -e 2014-04-27 -f 45 tecmint
 insert that user’s full name, Manis Khurana
 # useradd -c "Manis Khurana" mansi
 add a user ‘tecmint‘ without login shell
 # useradd -s /sbin/nologin tecmint
 create a user ‘ravi‘ with home directory ‘/var/www/tecmint‘, default shell /bin/bash and adds extra information about user
 # useradd -m -d /var/www/ravi -s /bin/bash -c "TecMint Owner" -U ravi
 # useradd -m -d /var/www/tarunika -s /bin/zsh -c "TecMint Technical Writer" -u 1000 -g 1000 tarunika
 disabling login shell to a user called ‘avishek‘
 # useradd -m -d /var/www/avishek -s /usr/sbin/nologin -c "TecMint Sr. Technical Writer" -u 1019 avishek
 useradd librenms -d /opt/librenms -M -r -s "$(which bash)"

 tecmint:x:504:504:tecmint:/home/tecmint:/bin/bash
    Username: User login name used to login into system. It should be between 1 to 32 charcters long.
    Password: User password (or x character) stored in /etc/shadow file in encrypted format.
    User ID (UID): Every user must have a User ID (UID) User Identification Number. By default UID 0 is reserved for root user and UID’s ranging from 1-99 are reserved for other predefined accounts. Further UID’s ranging from 100-999 are reserved for system accounts and groups.
    Group ID (GID): The primary Group ID (GID) Group Identification Number stored in /etc/group file.
    User Info: This field is optional and allow you to define extra information about the user. For example, user full name. This field is filled by ‘finger’ command.
    Home Directory: The absolute location of user’s home directory.
    Shell: The absolute location of a user’s shell i.e. /bin/bash.

 userdel member2 # deletes the account but leaves the user’s home directory intact
 userdel -f member2 # the -f option forces account deletion and fle removal under some circumstances
 userdel -r member2 # delete user account member2, the -r option to userdel causes it to delete the user’s home directory and mail spoo

 #disable root user

 sudo passwd -l root #lock the password for the root user
 sudo usermod -L root #lock the password for the root user
 sudo passwd -d root # remove the password of the account,can not unlock a locked account if it doesn't have any password

 #Enable Root Login
 sudo passwd root #set a new password for the account
 sudo passwd -u root #unlock the root user,unlock the password
 su root

 usermod -L testuser # lock user account / disable user account
 cat /etc/shadow | grep testuser # verify locked user account, exclamation mark "!"
 sudo passwd -S testuser #verify locked user account
 passwd --status testuser #verify locked user account
 usermod -U member1 # unlock user account
 passwd -u user1 #unlock the password

 # configure user default shell
 usermod -s /bin/bash member1
 # add the user geek to the group sudo
 usermod -a -G sudo geek 

 useradd -ms /bin/bash newuser #creates a home directory for the user and ensures that bash is the default shell
 useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1001 ubuntu

 ------------------------------------------------------------------------------------------
 #use useradd instead of its interactive wrapper adduser

 # create new user, new group with the same name
 sudo adduser sdn --system --group
 make sure username is added to the group libvirtd
 $ sudo adduser `id -un` libvirtd
 $ sudo adduser  $(id -un) libvirtd


 ------------------------------------------------------------------------------------------
 #edit the managers entry in /etc/group to make sally, tom, and dale  members of the group managers (GID 501),the
 #order of users in the comma-separated user list is unimportant
 managers:x:501:dale,sally,tom 

 cat /etc/group
 cat /etc/gshadow
 cut -d: -f1 /etc/group | sort
 $ groupadd admins #add new group
 $ sudo groupdel sshusers
 $ useradd -G admins member1
 sudo usermod -aG docker ${USER} #Add current logged in user to the docker group.
 sudo usermod -aG docker $(whoami) #Add current logged in user to the docker group.
 gpasswd -a devops1 sudo #Adding user devops1 to group sudo
 gpasswd -d devops1 sudo #Removing user devops1 from group sudo

 #list groups on Linux,/etc/group file
 $ getent group 
 $ getent group sudo # sudo group has ID of 27 and has ubuntu,barak,packer as members
 sudo:x:27:ubuntu,barak,packer
 $ cat /etc/group | awk -F: '{print $1}'
 $ cat /etc/group | cut -d: -f1


 $ groups # list the groups for the current logged user
 $ groups barak #get a list of groups a specific user is in

 whoami
 grep "^$(whoami):" /etc/subuid 
 grep "^$(whoami):" /etc/subgid
 id -u print only the effective user ID
 id -Gn
 id vagrant

 # The newgrp command allows a user to override the current primary group.
 # newgrp can be handy when you are working in a directory where all files must have the same group ownership
 newgrp microk8s #When you run the command, the system places you in a new shell and changes the name of your real group to the group specified with the Group parameter
 #If you’re listed as member of the group and the group has a password assigned, then you won’t be prompted
 #If there is no group password set, and the user is not listed as a member of the group, the user will be denied access
 #you are not a member of the developers group and not a root, then you’ll be prompted for a group password if your account doesn’t have a password assigned
 #If you’re root, then no prompted is presented.	
 newgrp developers 
 newgrp - developers #log in to the group developers
 sudo groupadd developers #create group developers
 sudo usermod -a -G developers richard
 sudo gpasswd developers

 #system account vs user account
 #System users will be created with no aging information in /etc/shadow
 #numeric identifiers are chosen in the SYS_UID_MIN–SYS_UID_MAX range defined in /etc/login.defs
 useradd --system appuser
 chage -l appuser
 useradd --system --shell=/usr/sbin/nologin <username> #create a system user (without home directory and login shell)
 useradd -r subversion #-r, --system  create a system account
 adduser -r -s /bin/nologin subversion #create system account -s /sbin/nologin to disable any login shell
 adduser --system --no-create-home --group yourusername
 adduser subversion --system --group
 adduser -r -s /bin/nologin subversion

 $ sudo groupadd --system --gid 1002 appuser
 $ cat /etc/group | grep appuser
 appuser:x:1002:
 $  sudo useradd --no-log-init --system --uid 1001 --gid 1002 appuser
 $ cat /etc/passwd | grep appuser
 appuser:x:1001:1002::/home/appuser:/bin/sh

 # list a group's members
 $ sudo lid -g sales
 $ cut -d: -f1,4 /etc/passwd | grep $(getent group sales | cut -d: -f3) | cut -d: -f1
 id username
 grep "docker" /etc/group
 grep -i --color 'root' /etc/group
 getent group -> List all groups
 cat /etc/group -> List all groups
 getent group vboxusers
 groups -> View the Groups a User Account
 groupmod -g 3000 foo -> assign a new GID to group called foo
 ------------------------------------------------------------------------------------------
 The command you've provided is a Linux shell command that utilizes the cp command to copy files and directories. Let's break down the components of the command:

    cp: This is the command itself, which stands for "copy." It's used to copy files or directories from one location to another.

    -vrbi: These are options or flags that modify how the cp command behaves:

        -v: Stands for "verbose." When this option is used, the cp command will provide more detailed output, showing the names of files as they are copied.

        -r: Stands for "recursive." This option is used when copying directories. It tells the cp command to copy not only the specified directory but also its contents and subdirectories.

        -b: Stands for "backup." With this option, the cp command will create backup copies of files that already exist in the destination directory before overwriting them.

        -i: Stands for "interactive." When this option is used, the cp command will prompt you for confirmation before overwriting files in the destination directory. This is a safety measure to prevent accidental data loss.

    /etc/skel/.: This is the source directory or file. In this case, it's the .skel directory located in the /etc directory. The /etc/skel directory is often used as a template for creating user home directories.

    ~/: This is the destination directory. The tilde (~) is a shorthand notation for the user's home directory. So, this part of the command specifies that the copied content should be placed in the user's home directory.

 The purpose of this command is to copy the contents of the /etc/skel directory (which often contains default settings and configurations for new user accounts) into the user's home directory. It uses various options like -v, -r, -b, and -i to control the copying process and handle situations where files already exist in the destination directory.

 To automatically answer "yes" to the interactive prompts and proceed with copying without manual confirmation, you can use the yes command in combination with the cp command. The yes command repeatedly outputs the letter "y" (or any other specified character) in response to prompts. Here's an example of how you could use it:

 sh

 yes | cp -vrbi /etc/skel/. ~/

 In this example, the yes command keeps providing "y" responses to the interactive prompts, effectively saying "yes" to each one. This allows the cp command to copy files without requiring manual confirmation.
 ------------------------------------------------------------------------------------------
 #Files in /etc/skel are copied from this directory to new users’ home directories by certain account-creation tools
 #The files should be in all new users’ home directories should reside in /etc/skel.
 “skeleton” directory is defined in /etc/default/useradd file.
 # ls -lart /etc/skel
 # ls -ldi /etc/skel
 33554552 drwxr-xr-x. 2 root root 62 Mar 24  2018 /etc/skel
 ------------------------------------------------------------------------------------------
 valid login shells
 # cat /etc/shells #list all available shells
 # chsh -l

 chsh -s /bin/ksh #Regular user can change their shell to the ksh

 $ ps -p $$
    PID TTY          TIME CMD
  56017 pts/0    00:00:00 bash
  
 grep tecmint /etc/passwd #view default login shell
 usermod --shell /bin/bash tecmint #change its login shell from /bin/sh to /bin/bash
 grep tecmint /etc/passwd #verify /bin/bash


 grep tecmint /etc/passwd #view default login shell
 chsh --shell /bin/bash tecmint #change its login shell from /bin/sh to /bin/bash
 grep tecmint /etc/passwd #verify /bin/bash

 $ sudo chsh -s /bin/ksh UserName #The superuser (root) changes the login shell for any account
 chsh -s /bin/ksh vagrant #set default login shell to /bin/ksh for vagrant user

 #open the /etc/passwd file, change manually
 vi /etc/passwd

 chsh vagrant -s /bin/rbash #change user vagrant's shell
 su - vagrant #login to activate shell

 whereis ksh
 grep --color ksh /etc/shells
 cat /etc/shells
 echo $SHELL

 ~ [23]$ ps -p $$
    PID TTY          TIME CMD
 193052 pts/1    00:00:00 ksh
 
 ~ [27]$ ps -hp $$ | awk '{print $5}'
 -ksh

 ~ [28]$ echo $0
 -ksh

 ~ [29]$ printf "%s\n" $0
 -ksh

 ~ [30]$ readlink /proc/$$/exe
 /usr/bin/ksh93

 awk -F: '/vagrant/ { print $7}' /etc/passwd #list user's shell type
 sudo ps -ef | egrep 'tty|pts' #Listing all shell types used by users
 $ w -h | awk '{print $2}' | xargs -L1 pgrep -oat

 verify user vagrant's bash
 # cat /etc/passwd | grep vagrant
 # echo $SHELL
 Reading Library editor
 $ cat /etc/inputrc
 keyboard bindings
 $ bind -v
 ------------------------------------------------------------------------------------------
  semicolon ";" multiple commands on the same line
  backaslash "\" run commands longer than one line
  press tab key or twice ESC key, command completion
  "./" run a command from pwd
 ------------------------------------------------------------------------------------------
 history file size setting
 $ cat /etc/profile
 $ echo $HISTSIZE
 $ echo $HISTFILE
 $ fc -l

 #Linux Command History with date and time, temporary
 HISTTIMEFORMAT="%d/%m/%y %H:%M "
 HISTTIMEFORMAT="%d/%m/%y %T "

 export HISTSIZE=0 #Disable the usage of history using HISTSIZE
 echo $HISTSIZE
 echo $HISTFILE
 export HISTCONTROL=ignoredups #Eliminate the continuous repeated entry from history using HISTCONTROL
 export HISTIGNORE="pwd:ls:ls -ltr:" #Ignore specific commands from the history using HISTIGNORE
 export HISTCONTROL=erasedups #Erase duplicates across the whole history using HISTCONTROL
 export HISTCONTROL=ignorespace #Force history not to remember a particular command using HISTCONTROL
 #  service httpd stop [Note that there is a space at the beginning of service,to ignore this command from history]

 history -c #Clear all the previous history

 # !ps #Execute previous command that starts with a specific word
 # !4 #Execute a specific command from history
 # !-1 #execute the second last command
 # !! #run the last executed command, or press CTRL+P
 # !dconf #re-run the last command with the keyword ‘dconf’ in it

 #lastb #shows users that failed to login,review the /var/log/btmp file (containing failed login attempts)

 #the login history of users
 last logins
 last -R #review the contents of the /var/log/wtmp binary file
 last | grep sysadmin
 last -f /var/log/btmp #Use the last command to view the btmp file
 last mark #pass the user name 
 last pts/0 #pass the tty
 last mark root pts/0 #specify multiple usernames and ttys
 last -p 2020-01-15 #find out who logged into the system on a specific date
 last -s 2020-02-13 -u 2020-02-18 #the -s (--since) and -t (--until) option to tell last to display the lines since or until the specified time
 last -F #y default, last doesn’t show the seconds and the year. Use the -F, --fulltimes option
 last -25 #last 25 logins
 last -i #IP address
 last -d #DNS address
 #the system last rebooted
 last reboot
  ------------------------------------------------------------------------------------------
  echo $PATH
  view all the env variables
  $ export -p
  $ set
  Set an Environment Variable
  $ export MYAPP=1
  holds the list of all directories that are searched by the shell when you type a command name
  $PATH
   
  system-wide
  $ cat /etc/profile
  single user
  $ cat .bash_profile
  # add PATH
 vi .bash_profile
 export PATH=$PATH:$HOME/Downloads/terraform
 system-wide prompt setting
 $ cat /etc/bashrc
 $ echo $PS1
 ------------------------------------------------------------------------------------------
 Display current libraries from the cache
 # ldconfig -p | head -5
 Display libraries from every directory
 ldconfig -v | head
 # cat /etc/ld.so.conf
 ------------------------------------------------------------------------------------------
 # number the lines in a file
 nl alphaservices | tee alphabetservices

 $ cat file1
 1. Asia:
 2. Africa:
 3. Europe:
 4. North America:
 Display the contents of file.txt in octal format (one byte per integer)
 $ od -b file1
 0000000 061 056 040 101 163 151 141 072 012 062 056 040 101 146 162 151
 0000020 143 141 072 012 063 056 040 105 165 162 157 160 145 072 012 064
 0000040 056 040 116 157 162 164 150 040 101 155 145 162 151 143 141 072
 0000060 012
 0000061
 Display the contents of file.txt in ASCII (character) format, with byte offsets displayed as hexadecimal.
 $ od -Ax -c file1
 000000   1   .       A   s   i   a   :  \n   2   .       A   f   r   i
 000010   c   a   :  \n   3   .       E   u   r   o   p   e   :  \n   4
 000020   .       N   o   r   t   h       A   m   e   r   i   c   a   :
 000030  \n
 000031
 ------------------------------------------------------------------------------------------
 echo -n "hola" | od -A n -t x1  | sed 's/ *//g'  | tr -d '\n' >> engineID_hex.txt #strips spaces  remove newlines
 ------------------------------------------------------------------------------------------
 # merge all files in the directory and split 
 ls | xargs cat | tee file1 | split -5
 # printing
 pr -h "title" file1
 list mounted file systems
 $ cat /etc/mtab

 split -l 4 index.txt split_file #Split file based on number of lines
 split index.txt -l 4 --verbose
 split -l 4 -a 4 index.txt #Change in suffix length. By default, the suffix length is 2
 split -l 4 -d index.txt #change the split files suffix to numeric
 split -l 4 index.txt split_index_ # create split output files with index suffix,
 split -l 4 -e index.txt #Avoid zero-sized split files
 split -n 3 index.txt #Create n chunks output files
 split -n 2 index.txt #Split the file into two files of equal length
 #split the file index.txt into separate files called indexaa, indexab, …..with each file containing 16 bytes of data
 split -b 16 index.txt index 
 split -b=1M -d  file.txt file --additional-suffix=.txt
 split -b 10M -d  system.log system_split.log


 ~$ cat test.txt | wc -l
 40
 ~$ split --numeric-suffixes=2 --additional-suffix=.txt -l 22 test.txt file
 $ ls -lai file*
 2139 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file02.txt
 5482 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file03.txt
 5483 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file04.txt
 18296 -rw-rw-r-- 1 vagrant vagrant 220 Mar 21 11:01 file05.txt
 $ cat file04.txt | wc -l
 12
 $ cat file05.txt | wc -l
 4
 -----------------------------------------------------------------------------------------------------
 u stands for user.
 g stands for group.
 o stands for others.
 a stands for all.

 same output:
 chmod -R 755 /var/www/html #change the permissions of all files and subdirectories under the /var/www/html directory to 755
 chmod +x somefile				(Based on umask value)
 chmod a+x somefile, chmod ugo+x somefile 	(Without considering umask value), add the execute permission for everyone

 chmod 644 a.txt
 stat -c %a a.txt # verify permission granted is 644
 ------------------------------------------------------------------------------------------
 Applying SUID Permission Numerically
 # chmod 4755 /bin/ping
 Removing SUID by Numerically
 chmod 0755 /bin/ping
 Applying SUID Permission to ping binary file Alphabetically
 # chmod u+s /bin/ping
 Removing SUID Permission
 chmod u-s /bin/ping
 ------------------------------------------------------------------------------------------
 # Applying SGID Permission
 chmod g+s /database/
 chmod 2775 database/
 # Remove SGID Alphabetically
 chmod g-s /database/
 chmod 0775 /database
 ------------------------------------------------------------------------------------------
 a sticky bit is now in place and only root, file or directory owners can rename and delete files
 # chmod +t /var/share/
 # ls -ld /var/share/
 drwxrwxrwt. 2 root root 4096 Mar  5 11:21 /var/share/

 chmod 0777 somefile (octal) 
 chmod 777 somefile (decimal)

 chmod 0710 mydir ; ls -ld mydir
 chmod 00710 mydir ; ls -ld mydir
 ------------------------------------------------------------------------------------------
 # quota settings
 sudo apt update ; sudo apt install quota -y
 quota --version
 find /lib/modules/`uname -r` -type f -name '*quota_v*.ko*'
 sudo mount -o remount /
 cat /proc/mounts | grep ' /
 sudo quotacheck -ugm /
 sudo quotaon -v /
 sudo setquota -u member1 200M 240M 0 0 /
 sudo quota -vs member1
 sudo setquota -t 864000 864000 /
 sudo repquota -s /
 ------------------------------------------------------------------------------------------
 # ldd (Unix) ldd (List Dynamic Dependencies)
 ldd /bin/ls
 # display unused direct dependencies
 ldd -u /bin/ping
 # more information
 ldd -v /bin/ping
 ------------------------------------------------------------------------------------------
 # rename root
 $ head -2 /etc/passwd
 root:x:0:0:root:/root:/bin/nologin
 rootmon:x:0:0:root:/root:/bin/bash
 $ sudo passwd rootmon
 $ su - rootmon
 # pwd
 /root

 #gain a root shell is by adding a new user to /etc/passwd who has the UID 0
 #any user with UID 0 is effectively root
 #root2 is the username
 #WVLY0mgH0RtUI is the encrypted password we want him to have. Unencrypted, the password is mrcake, in this case
 #0:0 means the user id and group id are both 0
 #root is a comment field
 #/root is the home directory
 #/bin/bash is the default shell
 root2:WVLY0mgH0RtUI:0:0:root:/root:/bin/bash

 #Administrative databases in Unix,getent – get entries from administrative database
    passwd – can be used to confirm usernames, userids, home directories and full names of your users
    group – all the information about Unix groups known to your system
    services – all the Unix services configured on your system
    networks – networking information – what networks your system belongs to
    protocols – everything your system knows about network protocols

 $ getent hosts # /etc/hosts file
 $ getent hosts vg-ubuntu-01 double-check which IPs this hostname points to
 $ getent networks #check the network and IP address of your system
 $ getent services 20 #Use “services” with the port number to find the service name and its protocol

 #List Users(system and normal users) on Linux using the /etc/passwd File, normal user has a real login shell and a home directory.
 awk -F: '{ print $1}' /etc/passwd
 cat /etc/passwd | awk -F: '{print $1}'
 awk -F: '{ print $1}' /etc/passwd | wc -l # get the # of users
 cut -d: -f1 /etc/passwd
 cat /etc/passwd | cut -d: -f1
 getent passwd # list users
 getent passwd | awk -F ":" '{print $1}'
 getent passwd | cut -d: -f1
 getent passwd # equivalent to cat /etc/passwd
 getent passwd rahul #details for a particular user
 getent passwd 0 #find a username by UID

 $ cut -d":" -f1 /etc/passwd #list all users

 #list normal user names
 awk -F: '{if($3 >= 1000 && $3 < 2**16-2) print $1}' /etc/passwd
 awk -F: '{if(($3 >= 500)&&($3 <65534)) print $1}' /etc/passwd
 awk -F: '{if(!(( $2 == "!!")||($2 == "*"))) print $1}' /etc/shadow 
 grep -E ":[0-9]{4,6}:[0-9]{4,6}:" /etc/passwd | cut -d: -f1
 $ getent passwd | awk 'NR==FNR { if ($1 ~ /^UID_(MIN|MAX)$/) m[$1] = $2; next }
 { split ($0, a, /:/);
  if (a[3] >= m["UID_MIN"] && a[3] <= m["UID_MAX"] && a[7] !~ /(false|nologin)$/)
    print a[1] }' /etc/login.defs -
 $ getent passwd | \
 nologin|false)> grep -vE '(nologin|false)$' | \
 : -v mi> awk -F: -v min=`awk '/^UID_MIN/ {print $2}' /etc/login.defs` \
 X/ {p> -v max=`awk '/^UID_MAX/ {print $2}' /etc/login.defs` \
 $3 >= > '{if(($3 >= min)&&($3 <= max)) print $1}' | \
 t -u> sort -u

 grep -E '^UID_MIN|^UID_MAX' /etc/login.defs #Each user has a numeric user ID called UID. If not specified automatically selected from the /etc/login.defs
 getent passwd {1000..60000} #list all normal users depending on UID_MIN/UID_MAX in /etc/login.defs
 eval getent passwd {$(awk '/^UID_MIN/ {print $2}' /etc/login.defs)..$(awk '/^UID_MAX/ {print $2}' /etc/login.defs)} | cut -d: -f1
 # generic,UID_MIN and UID_MIN values may be different, 
 eval getent passwd {$(awk '/^UID_MIN/ {print $2}' /etc/login.defs)..$(awk '/^UID_MAX/ {print $2}' /etc/login.defs)}


 awk -F ":" '{print $5}' /etc/passwd #print the fifth field
 getent passwd $UID| awk -F ":" '{print $5}'
 GECOS fields (which stands for "General Electric Comprehensive Operating System")
 username:password:userid:groupid:gecos:home-dir:shell
 GECOS are divided as:
 :FullName,RoomAddress,WorkPhone,HomePhone,Others:

 sally:x:0:529:Sally Jones:/home/myhome:/bin/passwd #might be used on, a Samba fle server or a POP mail server to enable users to change their passwords via SSH without granting login shell access.
 ------------------------------------------------------------------------------------------
 # enable the root account
 sudo passwd root
 ------------------------------------------------------------------------------------------
 # send an email from command line
 mail -s “Hello world” [email protected]
 echo “This will go into the body of the mail.” | mail -s “Hello world” [email protected]
 df -h | mail -s “disk space report” [email protected]
 ------------------------------------------------------------------------------------------
 # check a file system for errors?
 fsck
 fsck.ext3
 fsck.nfs
 fsck.ext2
 fsck.vfat
 fsck.reiserfs
 fsck.msdos

 In order to run fsck on the root partition, the root partition must be mounted as readonly
 ------------------------------------------------------------------------------------------
 # list of drives that are mounted at boot
 /etc/fstab
 # runs as a daemon and typically has PID 1
 # change the default runlevel upon boot up.
 /etc/inittab
 chkconfig --list #list of all runlevels and services used by them
 chkconfig vnicen.sh off #Ensure that the vnicen does not start upon reboot
 chkconfig --list | grep vnicen
 chkconfig --level 2345 sshd on #enable sshd startup
 syslogd # daemon is responsible for tracking events on the system
 # set which window man-ager you want to use when logging in to X from that account
 # edit in your home directory to change which window manager you want to use
 ~/.xinitrc 
 ------------------------------------------------------------------------------------------
 find the number of processing units (CPU) available on a system
 nproc
 nproc --all
 echo "Threads/core: $(nproc --all)"

 lscpu #the number of physical CPU cores
 lscpu | egrep 'Model name|Socket|Thread|NUMA|CPU\(s\)'
 lscpu -p

 grep 'model name' /proc/cpuinfo | wc -l
 grep 'cpu cores' /proc/cpuinfo | uniq
 echo "CPU threads: $(grep -c processor /proc/cpuinfo)"
 cat /proc/cpuinfo
 grep -c ^processor /proc/cpuinfo 
 cat /proc/cpuinfo | grep 'core id' #get the actual number of cores
 getconf _NPROCESSORS_ONLN && echo "Number of CPU/cores online at $HOSTNAME: $(getconf _NPROCESSORS_ONLN)"
 ------------------------------------------------------------------------------------------
 $ seq -s";" -w 1 12
 01;02;03;04;05;06;07;08;09;10;11;12

 # write dummy lines into a file
 $ seq -s ' ' 23 > file && cat file
 # read dummy lines from a file
 $ awk '(NR % 6 == 1) {print; for(i=1; i<6 && getline ; i++) { print }; printf "\n"}' RS=' ' ORS=' ' file
 1 2 3 4 5 6 
 7 8 9 10 11 12 
 13 14 15 16 17 18 
 19 20 21 22 23

 # Create a dummy file
 echo -e "1\n2\n3\n4" > testfile.txt

 $ echo "3997e1" > ids.txt #write
 $ echo "45697676107" >> ids.txt #append
 ------------------------------------------------------------------------------------------
 protection from inadvertently overwriting files when copying
 ~/.bashrc
 alias cp='cp -i'
 ------------------------------------------------------------------------------------------
 # list tree structure of files and folders

 $ ls -R
 .:
 create_folders_files.sh  folder_1  folder_2  folder_3

 ./folder_1:
 file_1.txt  file_2.txt  file_3.txt  file_4.txt

 ./folder_2:
 file_1.txt  file_2.txt  file_3.txt  file_4.txt

 ./folder_3:
 file_1.txt  file_2.txt  file_3.txt  file_4.txt


 $ find .
 .
 ./folder_2
 ./folder_2/file_1.txt
 ./folder_2/file_2.txt
 ./folder_2/file_3.txt
 ./folder_2/file_4.txt
 ./create_folders_files.sh
 ./folder_1
 ./folder_1/file_1.txt
 ./folder_1/file_2.txt
 ./folder_1/file_3.txt
 ./folder_1/file_4.txt
 ./folder_3
 ./folder_3/file_1.txt
 ./folder_3/file_2.txt
 ./folder_3/file_3.txt
 ./folder_3/file_4.txt


 $ sudo tree -d /var/log/ --du -sch
 /var/log/
 ├── [4.0K]  dist-upgrade
 ├── [4.0K]  fsck
 ├── [4.0K]  lxd
 ├── [4.0K]  apt
 └── [4.0K]  unattended-upgrades
 $ sudo tree /var/log/ --du -h

 $ sudo tree -a  /var/log #display hidden files
 $ tree -daC
 $ tree -f #view the full path for each directory and file
 $ sudo tree -f -L 3
 $ sudo tree -f -P cata* #only list files that match cata*, so files such as Catalina.sh, catalina.bat, etc
 $ tree -P "*.log"
 $ sudo tree -f -I *log /var/log #-I option,display all the files that do not match the specified pattern
 $ sudo tree -d -I *log /var/log 
 $ tree -I "*.log"
 $ sudo tree -f --prune #prune empty directories from the output 
 $ sudo tree -f -p #-p which prints the file type and permissions for each file
 $ sudo tree -f -pug #print the username,the group name
 $ sudo tree -f -pugs #print the size of each file in bytes along with the name using the -s option
 $ sudo tree -f -pugh #human-readable format, use the -h flag
 $ sudo tree -f -pug -h -D  #display the date of the last modification time for each sub-directory or file
 $ tree -d -L 3 # the depth of directory tree in output
 tree -vr #sort the files from Z-A
 $ tree -L 2
 tree -J #the output is in JSON format
 $ sudo tree -o direc_tree.txt
 ------------------------------------------------------------------------------------------
 ipcs (InterProcess Communication System) provides a report on the semaphore, shared memory & message queue
 ipcs -u
 ipcs -m
 ------------------------------------------------------------------------------------------
 nslookup github.com
 nslookup 140.82.118.4
 nslookup -query=mx github.com
 nslookup -query=ns github.com
 nslookup -query=any github.com
 nslookup -query=soa github.com
 nslookup -query=soa port=54 github.com
 nslookup -debug github.com
 ------------------------------------------------------------------------------------------
 The command shell interprets the && as the logical AND.the second command will be executed only when the first one has been succcefully executed
 A double ampersand && in Bash means AND and can be used to separate a list of commands to be run sequentially.
 Commands separated by a double ampersand && are to be run synchronously, with each one running only if the last did not fail (a fail is interpreted as returning a non-zero return status).

 && AND – execute both, return true of both succeed
 ; sequential execution, return status is that of the last in the list 

 $ mkdir /workspace ; mkdir /entrypoint
 mkdir: cannot create directory ‘/workspace’: Permission denied
 mkdir: cannot create directory ‘/entrypoint’: Permission denied
 $ mkdir /workspace && mkdir /entrypoint
 mkdir: cannot create directory ‘/workspace’: Permission denied
 ------------------------------------------------------------------------------------------
 mkdir -p first/second/third #If the first and second directories do not exist, mkdir creates these directories
 mkdir -m a=rwx first/second/third #set the file modes, i.e. permissions, etc
 ------------------------------------------------------------------------------------------
 disable and stop service
 $ sudo systemctl disable --now zabbix-server.service
 enable and start service
 $ sudo systemctl enable --now zabbix-server.service # enable and start the service

 chkconfig tgtd on #configure it to start Automatically while system start-up
 chkconfig --list tgtd #verify that the run level configured correctly for the tgtd service
 chkconfig --list #shows SysV services only and does not include native systemd services.
 chkconfig | grep snmpd
 systemctl list-units --all
 systemctl list-unit-files
 systemctl list-units --all --state=inactive
 systemctl list-units --type=service #only active service units

 systemctl cat sshd.service #Displaying a Unit File
 sudo systemctl edit nginx.service
 sudo systemctl edit --full nginx.service
 sudo systemctl daemon-reload

 systemctl list-unit-files --type=target
 systemctl list-units --type=target

 systemctl list-dependencies sshd.service #Displaying Dependencies
 systemctl show sshd.service #Checking Unit Properties
 systemctl show sshd.service -p Conflicts #display a single property,pass the -p flag with the property name
 sudo systemctl mask nginx.service #the ability to mark a unit as completely unstartable, automatically or manually, by linking it to /dev/null. This is called masking the unit

 ------------------------------------------------------------------------------------------
 ls -laZ ~/.ssh
 # change the security context to system_u:object_r:usr_t:s0
 chcon -R -v system_u:object_r:usr_t:s0 ~/.ssh/
 ------------------------------------------------------------------------------------------
 problem: 
 never edit directly '/etc/sudoers' file
 $ sudo visudo
 >>> /etc/sudoers: syntax error near line 28 <<<
 sudo: parse error in /etc/sudoers near line 28
 sudo: no valid sudoers sources found, quitting

 fix:
 pkexec visudo
 #includedir /etc/sudoers -> #includedir /etc/sudoers.d #change last line

 sudo mount /dev/sda1 /mnt #mount the installed Ubuntu system's root filesystem
 sudo visudo -f /mnt/etc/sudoers #edit the installed system's sudoers file
 pkexec visudo -f /etc/sudoers.d/filename #edit configuration files
 ------------------------------------------------------------------------------------------
 problem: sleep: invalid time interval `2\r'
 fix: sudo cat test.sh | sudo tr -d '\r' | sudo tee test2.sh
 ------------------------------------------------------------------------------------------
 $ diff 1.txt 2.txt # display the differences in the files by comparing the files line by line
 $ diff -c 1.txt 2.txt
 $ diff 1.txt 2.txt -u
 $ diff 1.txt 2.txt -i
 $ diff 1.txt 2.txt --color
 $ diff 1.txt 2.txt -s
 $diff -i test_file_1.txt test_file_2.txt #ignoring the case sensitivity
 $diff -y -W 60 test_file_1.txt test_file_2.txt #view the difference,The “-W” indicates the width between the content of two files
 $diff -q test_file_1.txt test_file_2.txt #“-q” option with the “diff” command gives you output in one line
 $diff -u test_file_1.txt test_file_2.txt #

 #write the difference between two files into a file
 diff a.txt b.txt|grep ">"|cut -c 3- > foo.txt

 #-q, --brief report only when files differ
 #-s, --report-identical-files report when two files are the same
 diff -sq /tmp/file1 /tmp/file2

 diff chap1.back chap1 #compare two files
 #If two lines differ only in the number of spaces and tabs between words, the diff -w command considers them to be the same.
 #compare two files compare two files while ignoring differences in the amount of white space
 diff  -w prog.c.bak prog.c 
 #create a file containing commands that the ed command can use to reconstruct one file from another
 #creates a file named new.to.old.ed that contains the ed subcommands to change chap2 
 #back into the version of the text found in chap2.old
 diff  -e chap2 chap2.old >new.to.old.ed 
 # in parentheses add 1,$p to the end of the editing commands sent to the ed editor. 
 #The 1,$p causes the ed command to write the file to standard output after editing it
 #then piped to the ed command (| ed), and the editor reads it as standard input
 #The - flag causes the ed command not to display the file size and other extra information 
 #because it would be mixed with the text of chap2.old.
 (cat new.to.old.ed ; echo '1,$p') | ed - chap2 >chap2.old
 #compare two text files containing UTF-8 characters and show the differences
 diff -W filecodeset=UTF-8,pgmcodeset=IBM-1047 myUtf8File01 myUtf8File02
 #compare two text files containing EBCDIC characters and show the differences
 diff -B myMisTaggedFile01 myMisTaggedFile02
 ------------------------------------------------------------------------------------------
 Local to Remote: rsync [OPTION]... -e ssh [SRC]... [USER@]HOST:DEST
 Remote to Local: rsync [OPTION]... -e ssh [USER@]HOST:SRC... [DEST]

 -v, –verbose – Using this option in the rsync command gives the status about transferred files.
 -vv – Usually, this option is used to get information about the skipped files during file transfer.
 -q, –quiet – This option simply suppress non-error messages.

 rsync -av --rsync-path="rsync --log-file=/tmp/rlog" source/ destination/ #enable error log for rsync
 #rsync exits with a non-zero code when the transfer fails,write details to log files.
 rsync -avz -e ssh [email protected]:/ /mybackup/ > /var/log/rsync.log 2>&1

 Problem:
 rsync: failed to set times on "some_dir: Operation not permitted (1)
 mkstemp "some_file" failed: Permission denied (13)
 Fix:
 if the user is 'abc' then 
 the destination directory should be 
 lrwxrwxrwx 1 abc abc 34 Jul 18 14:05 Destination_directory
 chown -R abc:abc Destination_directory

 rsync -aEim --delete /path/to/remote/ /path/to/local/ # rsync output to stdout with the -i flag

 #Only files that rsync has fully successfully transferred are removed.
 rsync -r -z -c --remove-source-files  /home/pi/queue [email protected]:/home/foobar 

 rsync -avz source destination #preserve permissions, ownership, and timestamp

 #When the trailing slash  "/" is omitted the source directory will be copied inside the destination directory
 #transfer the local directory to the directory on a remote machine
 $ rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /home/filerunner/dir1 vg-ubuntu-02:/tmp
 $ ls /tmp/dir1
 a.txt

 #When the source directory has a trailing slash  "/", rsync will copy only the contents of the source directory to the destination directory
 $ rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /home/filerunner/dir1/ vg-ubuntu-02:/tmp
 $ ls /tmp
 a.txt

 #dry run mode,
 rsync -azhv -e "ssh -p 2212" --dry-run /home/bob/test_219 

 #find out if the files are in sync, without actually doing a sync dry run mode
 #-c, --checksum - skip based on checksum, not mod-time & size
 #-r, --recursive - recurse into directories
 #-n, --dry-run - perform a trial run with no changes made
 #does not show anything if server2 has more files then server1
 rsync -n -avrc /abc/home/sample1/* server2:/abc/home/sample2/
 #dry-run mode verify
 rsync -avzrch --progress --exclude=lost+found/ -e "ssh -i /home/vagrant/privatekey" --delete user@server:/mnt/files/ /mnt/disk1/ > /tmp/rsync_out 2>&1


 #find out if the files are in sync, without actually doing a sync dry run mode
 #--delete is needed to show if a file exists on server 2 but not server 1
 rsync -n -avr --size-only --delete /abc/home/sample1/ server2:/abc/home/sample2/

 #find out if the files are in sync, without actually doing a sync dry run mode
 #Without --dry-run, it will automatically copy files of different sizes 
 #if the sizes are identical, checksum them and copy if different
 #The delete option will remove files from the target if not present on the source
 rsync -cr --delete --dry-run source/ target/ > output_file 2>&1 &
 #--size-only             skip files that match in size, no checksum
 rsync -n -avr --size-only --delete /abc/home/sample1/ server2:/abc/home/sample2/

 rsync -a -e "ssh -p 3322" /home/linuxize/images/ [email protected]:/var/www/images/ #if SSH is listening on port 3322

 #transfer a single file /opt/file.zip from the local system to the /var/www/ directory on the remote system with IP 12.12.12.12
 #If the file exists on the remote server it is overwritten
 rsync -a /opt/file.zip [email protected]:/var/www/ 
 #save the file under a different name
 rsync -a /opt/file.zip [email protected]:/var/www/file2.zip

 #transfer data from a remote to a local machine
 rsync -a [email protected]:/var/www/file.zip /opt/

 #synchronize the local and remote directory
 rsync -a /home/linuxize/images/ [email protected]:/var/www/images/

 #use the --delete option if you want to synchronize the local and remote directory
 #delete files in the destination directory if they don’t exist in the source directory.
 rsync -a --delete /home/linuxize/images/ [email protected]:/var/www/images/


 #the “-r” option for “recursive” and the “-a” option for “all” (otherwise non-regular files will be skipped) 
 #copy the “/etc” folder to the “/etc_backup” of the remote server
 #with the “devconnected” username to server 192.168.178.35/24
 rsync -ar /etc [email protected]:/etc_backup

 #Similarly,copy the content of the “/etc/ directory rather than the directory itself
 rsync -ar /etc/* [email protected]:/etc_backup/ 

 # taggged with the current date
 rsync -ar /etc/* [email protected]:/etc_backup/etc_$(date "+%F")

 #from local to remote server with private key
 rsync -auvz -e "ssh -i private-key-file" source destination #Using rsync With SSH and Private Key 
 rsync -auvz -e "ssh -i /home/yourUserName/.ssh/yourUserName-rsync-key" junk.txt [email protected]
 rsync -avzhe ssh backup.tar.gz [email protected]:/backups/
 rsync -avzhe ssh --progress /root/rpmpkgs [email protected]:/root/rpmpkgs
 
 #from remote to local server
 rsync -avzh [email protected]:/root/rpmpkgs /tmp/myrpms
 rsync -avze ssh --include 'R*' --exclude '*' [email protected]:/var/lib/rpm/ /root/rpm
 #exclude lost+found dir
 rsync --archive --no-compress --delete-before --info=progress2 --human-readable --exclude=lost+found/ /mnt/backup/ /mnt/backup-2/

 #run rsycn on the background
 rsync -avze ssh --include 'R*' --exclude '*' [email protected]:/var/lib/rpm/ /root/rpm > rsync.out 2>&1 &
 tail -f rsync.out
 ------------------------------------------------------------------------------------------
 #copy the “/etc” directory to a backup server located at 192.168.178.35 in the “/etc_backup” folder
 scp -r /etc [email protected]:/etc_backup/
 # taggged with the current date
 scp -r /etc [email protected]:/etc_backup/etc_$(date "+%F")

   scp [email protected]:foobar.txt /some/local/directory-> Copy the file "foobar.txt" from a remote host to the local host
   scp file.txt [email protected]:/remote/directory/newfilename.txt # save the file under a different name,Omitting the filename from the destination location copies the file with the original name.
   scp foobar.txt [email protected]:/some/remote/directory -> Copy the file "foobar.txt" from the local host to a remote host
   scp [email protected]:/some/remote/directory/foobar.txt [email protected]:/some/remote/directory/ ->Copy the file "foobar.txt" from remote host "rh1.edu" to remote host "rh2.edu"
   scp -P 2322 file.txt [email protected]:/remote/directory #the remote host is listening on a port other than the default 22 
   scp -r /local/directory [email protected]:/remote/directory #copy a directory from a local to remote system,use the -r flag for recursive
   
   # don’t have to log in to one of the servers to transfer files from one to another remote machine.
   #copy the file /files/file.txt from the remote host host1.com to the directory /files on the remote host host2.com
   scp [email protected]:/files/file.txt [email protected]:/files
   scp -3 [email protected]:/files/file.txt [email protected]:/files #route the traffic through the local host (machine on which the command is issued), use the -3 option
 ------------------------------------------------------------------------------------------
 #cp interpret main as a directory to place scala,doesn't exist, cp will throw an error.
 cp -av  /home/jake/transit/scalaProjects/scalaML/src/main/scala -t /home/jake/project/__workspace/scalaProjects/scalaML/src/main/

 #Copy Directory Content Recursively	
 cp -R bashdir bashdir-bck
 cp -R /etc/* /etc_backup
 cp -R /etc/* /home/* /backup_folder # copy the “/etc” directory and “/home” directory.

 #copies the folder Misc and all its contents (the -r, or "recursive," option indicates the contents as well as the folder or file itself) into the folder /media/clh/4388-D5FE
 cp -r Misc /media/clh/4388-D5FE #
 #copy over only the new files,use the "update" and "verbose" options
 cp -ruv Misc /media/clh/4388-D5FE
 # a file called test1.py, which is the original, and another called test1.py.~1~, which is the backup file.
 cp --force --backup=numbered test1.py test1.py 

 cp /home/usr/dir/{file1,file2,file3,file4} /home/usr/destination/ #copy multiple files 
 cp -rp /copying/from/{folder1/,folder2/,folder3/} path/to/folder #p is for copying the folder permission
 cp /home/usr/dir/file{1..4} /tmp #if the all the files have the same prefix but different endings

 ------------------------------------------------------------------------------------------
 #brace expansions

 $ sudo cp ~/a.txt{,.orig} #backup of file in the current directory, copy shortcut
 $ ls
 a.txt  a.txt.orig

 $ echo g{et,ot,it}em
 getem gotem gitem
 $ echo {00..8..2} #loop with increase 2
 00 02 04 06 08
 $ echo {D..T..4} #loop with increase 4
 D H L P T

 $ mv error.log{,.OLD} #expands to "mv error.log error.log.OLD")
 ------------------------------------------------------------------------------------------
 #rename multiple files to change the extensions
 $ mv *.txt *.tsv
 mv: target 'd.tsv': Not a directory

 #rename all the files with a .txt extension to .fg extension in the given directory
 find . -type f -name "*.txt" -exec sh -c 'mv "$1" "${1%.txt}.fg"' _ {} \;

 #bash script
 for file in /path/to/directory/*.txt; do
  mv "$file" "${file%.txt}.fg"
 done

 # one liner version of the bash script above
 find /path/to/directory -type f -name "*.txt" -exec bash -c 'mv "$0" "${0%.txt}.fg"' {} \;
 ------------------------------------------------------------------------------------------
 cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found
 cmp --silent file1 file2 && echo 'SUCCESS: Files Are Identical' || echo 'Files Are Different'
 cmp --silent $old $new || echo "files are different"

 #if= defines the source drive and of= defines the file or location where data saved
 # dd if=/dev/sda of=/dev/sdb
 # dd if=/dev/sda of=/home/username/sdadisk.img #create an .img archive of the /dev/sda drive and save it to the home directory
 # dd if=/dev/sda2 of=/home/username/partition2.img bs=4096
 #if= takes the image to restore, and of= takes the target write the image
 # dd if=sdadisk.img of=/dev/sdb
 create a compressed image of a remote drive using SSH and save the resulting archive to local machine
 # ssh [email protected] "dd if=/dev/sda | gzip -1 -" | dd of=backup.gz
 dd  if=text.ascii of=text.ebcdic conv=ebcdic #convert an ASCII text file to EBCDIC
 #convert the variable-length record ASCII file /etc/passwd to a file of 132-byte fixed-length EBCDIC records
 dd  if=/etc/passwd  cbs=132  conv=ebcdic  of=/tmp/passwd.ebcdic 
 #convert the 132-byte-per-record EBCDIC file to variable-length ASCII lines in lowercase
 dd  if=/tmp/passwd.ebcdic  cbs=132  conv=ascii  of=/tmp/passwd.ascii
 #copy blocks from a tape with 1KB blocks to another tape using 2KB blocks
 dd  if=/dev/rmt0  ibs=1024  obs=2048  of=/dev/rmt1
 ls -l | dd  conv=ucase #displays a long listing of the current directory in uppercase.
 dd if=/dev/zero of=/dev/sda1 #Wiping disks with dd,writing zeros
 # dd if=/dev/urandom of=/dev/sda1 #Wiping disks with dd,writing random characters
 # dd if=/dev/urandom | pv | dd of=/dev/sda1 #Monitoring dd operations,Pipe Viewer (sudo apt install pv on Ubuntu) 

 #create a large file of random conten
 $ dd if=/dev/urandom of=/tmp/file1 count=1K bs=1MB 
 #copy file1 to file2, and append different characters to each file
 $ cp /tmp/file1 /tmp/file2
 $ echo 1 >> /tmp/file1
 $ echo 2 >> /tmp/file2
 #use the time command to measure the time taken
 $ time cmp -s /tmp/file1 /tmp/file2 
 $ time diff -sq /tmp/file1 /tmp/file2 
 $ time sha1sum /tmp/file1
 $ time sha1sum /tmp/file2

 #Empty File Content by Redirecting to Null
 # > access.log
 # : > access.log #: is a shell built-in command
 # true > access.log
 # cat /dev/null > access.log
 # cp /dev/null access.log
 # dd if=/dev/null of=access.log
 # echo "" > access.log
 # echo > access.log
 # echo -n "" > access.log #use the flag -n which tells echo to not output the trailing newline
 # truncate -s 0 access.log


 dd if=/dev/urandom of=test.file bs=1M count=100 ; time diff -q test.file test.copy && echo diff true || echo diff false ; \
 time cmp -s test.file test.copy && echo cmp true || echo

 vagrant@lampstack-01:~$ echo "file" > file1.txt
 vagrant@lampstack-01:~$ cp file1.txt file2.txt
 vagrant@lampstack-01:~$ cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found
 vagrant@lampstack-01:~$ sudo cmp file1.txt file2.txt
 vagrant@lampstack-01:~$ echo "identical file1" >> file1.txt
 vagrant@lampstack-01:~$ cat file1.txt
 file
 identical file1
 vagrant@lampstack-01:~$ cat file2.txt
 file
 vagrant@lampstack-01:~$ cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found
 cmp: EOF on file2.txt after byte 5, line 1
 vagrant@lampstack-01:~$ diff file1.txt file2.txt
 2d1
 < identical file1
 ------------------------- -----------------------------------------------------------------
 # approval trick
 $ yes | sudo yum install puppet
 # user has to type 'y' for each query
 $ yes | rm -ri test
 ------------------------- -----------------------------------------------------------------
 #oracle java download

 wget --no-cookies \
 --no-check-certificate \
 --header "Cookie: oraclelicense=accept-securebackup-cookie" \
 "https://download.oracle.com/otn-pub/java/jdk/13.0.1+9/cec27d702aa74d5a8630c65ae61e4305/jdk-13.0.1_linux-x64_bin.tar.gz" \
 -O jdk-7-linux-x64.tar.gz


 curl --silent example.com | sha256sum
 curl --silent --output - example.com | sha256sum #the o flag is redundant since the output is piped to bash (for execution) - not to a file
 #-L in case the page has moved curl will redirect the request to the new address
 #-o output to a file instead of stdout (usually the screen)
 curl -LO http://example.com/

 curl -LO -H "Cookie: oraclelicense=accept-securebackup-cookie" \
 https://download.oracle.com/otn-pub/java/jdk/13.0.1+9/cec27d702aa74d5a8630c65ae61e4305/jdk-13.0.1_linux-x64_bin.tar.gz
 ------------------------------------------------------------------------------------------

 ------------------------------------------------------------------------------------------
  hostnamectl set-hostname vg-checkmk-client
  echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" |sudo tee -a /etc/hosts
  echo "nameserver 8.8.8.8" |sudo tee -a /etc/resolv.conf

 problem:
 #append text to a file when using sudo
 sudo echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" >> /etc/hosts
 -bash: /etc/hosts: Permission denied
 fix:
 #running bash/sh shell with root privileges and redirection took place in that shell session
 $ sudo sh -c 'echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" >> /etc/hosts'
 $ sudo sh -c 'echo "JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile'
 $ sudo bash -c 'echo "JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile'
 $ sudo tee -a /etc/profile.d/java.sh << 'EOF'
 # configures JAVA
 JAVA_HOME=/usr/lib/jvm/java-8-oracle
 export JAVA_HOME
 export PATH=$PATH:$JAVA_HOME/bin
 EOF
 $ sudo sed -i '$a something' /etc/config_file
 ------------------------------------------------------------------------------------------	  
 #File Creation Times
 vagrant@lampstack-01:/tmp/nexus$ df -h
 Filesystem                    Size  Used Avail Use% Mounted on
 udev                          205M     0  205M   0% /dev
 tmpfs                          48M  7.8M   41M  17% /run
 /dev/mapper/vagrant--vg-root   62G  4.3G   55G   8% /
 tmpfs                         240M     0  240M   0% /dev/shm
 tmpfs                         5.0M     0  5.0M   0% /run/lock
 tmpfs                         240M     0  240M   0% /sys/fs/cgroup
 vagrant                       420G  375G   46G  90% /vagrant
 tmpfs                          48M     0   48M   0% /run/user/1000
 vagrant@lampstack-01:/tmp/nexus$ ls -i Dockerfile
 3808483 Dockerfile
 vagrant@lampstack-01:/tmp/nexus$ sudo debugfs -R 'stat <3808483>' /dev/mapper/vagrant--vg-root
 debugfs 1.44.6 (5-Mar-2019)
 vagrant@lampstack-01:/tmp/nexus$ vagrant@lampstack-01:/tmp/nexus$ sudo debugfs -R 'stat <3808483>' /dev/mapper/vagrant--vg-root | grep crtime
 debugfs 1.44.6 (5-Mar-2019)
 crtime: 0x5e4f0f92:b6a11330 -- Thu Feb 20 23:00:34 2020
 ------------------------------------------------------------------------------------------
 #how to view gz gunzip files

 zcat test2.txt.gz #see the content of the compressed file
 zcat test.txt.gz test2.txt.gz #see the content of the compressed files,multiple inputs
 zcat test2.txt #uncompress files that have the correct magic number whether they have a .gz suffix or not
 zcat test2.txt.gz | more
 zcat test2.txt.gz | less
 zmore test2.txt.gz
 zless test2.txt.gz
 zless -f test2.txt #display file contents in output whether or not the file is compressed

 # search string in multiple .gz file
 zgrep -i -H "pattern match" /somedir/filename*.gz
 zgrep -i -H "pattern match" /somedir/filename*.gz | grep "pattern 2nd"
 find /somedir/ -name "log.202011917*.gz" -exec zgrep "somestring"  \{\} \;
 find /somedir/ -name 'log.202011917' -print0 | xargs -0 zgrep "somestring" #prints file path inc. folder
 ------------------------------------------------------------------------------------------
    #Hosts File
    Windows 10 - "C:\Windows\System32\drivers\etc\hosts"
    Linux - "/etc/hosts"
    Mac OS X - "/private/etc/hosts"
 ------------------------------------------------------------------------------------------    
 type -a ll #if command is an alias or not
 type -p dash
 ------------------------------------------------------------------------------------------   
 ------------------------------------------------------------------------------------------
 echo $? #get the exit status of the previously executed command, http://www.tldp.org/LDP/abs/html/exitcodes.html
 echo $? # Expands to the exit status (exit code) of the most recently executed foreground pipeline,return the exit status of last command
 exits with a status code 0 #success
 exits with a status code 1 #failure

 command && echo "success: $?" || echo "fail: $?" #test if the command failed
 if [ $? -eq 0 ]; then echo "success: $?"; fi

 $ echo "hola el mundo" > file.txt
 $ cat file.txt && echo "success: $?" || echo "fail: $?"
 hola el mundo
 success: 0
 $ cat filedoesnotexist.txt && echo "success: $?" || echo "fail: $?"
 cat: filedoesnotexist.txt: No such file or directory
 fail: 1

 cat 'doesnotexist.txt' 2>/dev/null || exit 0 #suppress exit status
 cat file.txt || exit 0

 #suppress the error silently
 $ cat filedoesnotexist.txt && echo "success: $?" || echo "fail: $?"
 cat: filedoesnotexist.txt: No such file or directory
 fail: 1
 $ cat filedoesnotexist.txt || true && echo $?
 cat: filedoesnotexist.txt: No such file or directory
 0

 $ fslint /tmp #lists the duplicate files
 $ rdfind /tmp #delete the duplicates,remove the newer files.
 $ rdfind -dryrun true /tmp #only report the changes
 $ rdfind -deleteduplicates true /tmp

 $ cksum *.html #computes checksums for files
 $ find . -name "*.html" -exec cksum {} \; #search files by name or type and run the cksum command.

 $ find . -name "not_existing_file" 
 $ echo $?
 -----------------------------------------------------------------------------------------------------
 bash -x # runs the script <file> with tracing of each command executed
 bash -x -c ls -lai #run a command in BASH, use -c option
 test -x <file> #tests whether <file> has execute permissions for the current user
 ------------------------------------------------------------------------------------------  
 #print variable
 export BRANCH_NAME="main"
 echo "BRANCH_NAME is..: $BRANCH_NAME"
 ------------------------------------------------------------------------------------------   
 yum/apt install chrony
 systemctl stop chronyd
 chronyd -q 'pool pool.ntp.org iburst'
 systemctl start chronyd
 chronyc tracking #verify
 systemctl restart chronyd ; watch chronyc tracking #realtime witnessing
 chronyc sources
 chronyc sources -v
 chronyc
 ------------------------------------------------------------------------------------------
 #concatenate strings
 export strservice="libvirtd"
 echo "${strservice} is still running."
 echo $(ls)
 echo "The date is $(date)"
 echo `pwd`
 echo $((3 + (4**3 /2))) #Direct calculation in the shell with echo and $(( )
 ------------------------------------------------------------------------------------------
 #nfs server
 exportfs -arv
 df -h

 cat /etc/exports
 # /mnt/nfs_share  client_IP_1 (re,sync,no_subtree_check)
 echo "$NFS_DIR  192.168.50.7(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
 echo "$NFS_DIR  192.168.50.8(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports

 #nfs client  mount -t nfs NFS_SERVER_IP:NFS_SERVER_DIR NFS_CLIENT_DIR
 mount -t nfs  10.20.20.8:/mnt/backups /mnt/backups
 umount /mnt/backups
 mount | grep nfs
 showmount -e
 ------------------------------------------------------------------------------------------
 #execute shell command produced using echo and sed
 echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1" | bash #
 bash -c "$(echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1")" #pass it as an argument to a shell
 eval "$(echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1")" #use the bash built-in eval
 ------------------------------------------------------------------------------------------
 $ mycommand="wc -l department.txt"
 $ eval $mycommand
 ------------------------------------------------------------------------------------------
 $ cat /proc/sys/net/ipv4/ip_local_port_range
 ------------------------------------------------------------------------------------------
 #redirection is done by the shell which doesn't has write permission
 sudo echo 1 > /proc/sys/net/ipv4/ip_forward
 bash: /proc/sys/net/ipv4/ip_forward: Permission denied

 echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward #resolution-1

 $ sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward' #resolution-2

 $ sudo -s #resoluton-3
 # echo 1 > /proc/sys/net/ipv4/ip_forward

 $ cat testscript.sh #resolution-4
 #!/bin/sh
 echo 1 > /proc/sys/net/ipv4/ip_forward

 sudo echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward > /dev/null #same
 $ sudo tee /proc/sys/net/ipv4/ip_forward > /dev/null << EOF	#same
 1
 EOF
 ------------------------------------------------------------------------------------------
 #source tar build, no version typing
 tar zfx rrdtool.tar.gz
 cd rrdtool-*
 ------------------------------------------------------------------------------------------
 echo 1 > /proc/sys/kernel/sysrq;echo f > /proc/sysrq-trigger;echo 0 > /proc/sys/kernel/sysrq #Trigger Out Of Memory (OOM) killer without reboot
 ------------------------------------------------------------------------------------------
 ls -la .mozilla/firefox #user preferences,browser cookies,bookmarks,browser cache content etc.
 ------------------------------------------------------------------------------------------
 #troubleshooting memory problems

 #Suddenly killed tasks are often the result of the system running out of memory, when the so-called Out-of-memory (OOM) killer steps in
 $ grep -i -r 'out of memory' /var/log/ #search the logs for messages of out of memory alerts

 $ free -m #megabytes
 $ free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'
 watch -n 5 -d '/bin/free -m'

 $ top
 $ top -i -u vagrant #ignore idle processes
 $ top -n 1 -o RES | grep kvm

 $ uptime
 #"system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads.
 #show three averages, for 1, 5, and 15 minutes
 #If the averages are 0.0, system is idle
 #If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing
 #If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.

 $ ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1
 $ ps -eL h -o state | egrep "R|D" | wc -l #The instantaneous number of such tasks
 #Linux load average,the instantaneous load of a system the number of tasks (processes and threads) 
 #that are willing to run at a given time t
 #either in state R or D, either actually running or blocked on some resource (CPU, IO, ...) waiting for an opportunity to run

 $ cat /proc/sys/vm/swappiness
 #The Linux kernel moves out pages which are not active or being used at the moment to swap space on the disk. This process is known as swappiness.
 #turn off swaping by changing t he value in /proc/sys/vm/swappiness to 0.The value ranges from 0 to 100 whe re 100 means aggressive swapping

 vmstat –a 1 99 #show memory usage information
 vmstat -n 1 # If 'si' and 'so' (stands for swapin and swapout) fields are always 0, then the system is currently not swapping

 $ grep DirectMap /proc/meminfo
 $ cat /proc/meminfo #Relevant fields from /proc/meminfo to match them against the output of free -k
    MemTotal — Total amount of physical RAM, in kilobytes.
    MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
    Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
    Cached — The amount of physical RAM, in kilobytes, used as cache memory.
    SwapCached — The amount of swap, in kilobytes, used as cache memory.
    Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
    Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
    HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
    LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
    SwapTotal — The total amount of swap available, in kilobytes.
    SwapFree — The total amount of swap free, in kilobytes.
    Dirty — The total amount of memory, in kilobytes, waiting to be written back to the disk. Writeback — The total amount of memory, in kilobytes, actively being written back to the disk.
    Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
    Slab — The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
    Committed_AS — The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory. PageTables — The total amount of memory, in kilobytes, dedicated to the lowest page table level.
    VMallocTotal — The total amount of memory, in kilobytes, of total allocated virtual address space.
    VMallocUsed — The total amount of memory, in kilobytes, of used virtual address space.
    VMallocChunk — The largest contiguous block of memory, in kilobytes, of available virtual address space.
    HugePages_Total — The total number of hugepages for the system. The number is derived by dividing Hugepagesize by the megabytes set aside for hugepages specified in /proc/sys/vm/hugetlb_pool. This statistic only appears on the x86, Itanium, and AMD64 architectures.
    HugePages_Free — The total number of hugepages available for the system. This statistic only appears on the x86, Itanium, and AMD64 architectures.
    Hugepagesize — The size for each hugepages unit in kilobytes. By default, the value is 4096 KB on uniprocessor kernels for 32 bit architectures. For SMP, hugemem kernels, and AMD64, the default is 2048 KB. For Itanium architectures, the default is 262144 KB. This statistic only appears on the x86, Itanium, and AMD64 architectures.

 Matching output of free -k to /proc/meminfo
 free output 	coresponding /proc/meminfo fields
 Mem: total 	MemTotal
 Mem: used 	MemTotal - MemFree
 Mem: free 	MemFree
 Mem: shared (can be ignored nowadays. It has no meaning.) 	N/A
 Mem: buffers 	Buffers
 Mem: cached 	Cached
 -/+ buffers/cache: used 	MemTotal - (MemFree + Buffers + Cached)
 -/+ buffers/cache: free 	MemFree + Buffers + Cached
 Swap: total 	SwapTotal
 Swap: used 	SwapTotal - SwapFree
 Swap: free 	SwapFree

 ------------------------------------------------------------------------------------------
 cat newpass.txt | chpasswd #update passwords in batch mode
 echo 'ubuntuser:ubuntupassword' | sudo chpasswd

 $cat > pass.txt
 user1:user1_password
 user2:user2_password
 user3:user3_password
 $chpasswd < file_name.txt

 CURRENT_USERS=$(who) #Assign Output of a Linux Command to a Variable
 ------------------------------------------------------------------------------------------
 ls -1 #list one file per line,vertically using the -1 switch
 ------------------------------------------------------------------------------------------
 sudo !! #Re-Run Last Executed Command as Root User, “sudo” followed by a space and two exclamation points. 
 ------------------------------------------------------------------------------------------
 #If set, sudo will only run when the user is logged in to a real tty
 #sudo can only be run from a login session and not via other means such as cron(8) or cgi-bin scripts
 #enable sudo within scripts

 sudo /usr/sbin/visudo

 $ cat /etc/sudoers
 Defaults:username !requiretty #Disable requiretty to particular user


 #sudoers file allows both artbristol and bob to execute /path/to/program as root from a script
 #artbristol needs no password whereas bob must have to enter a password
 artbristol ALL = (root) NOPASSWD: /path/to/program
 bob ALL = (root) /path/to/program
 Defaults!/path/to/program !requiretty

 #allows artbristol to run /path/to/program --option in a script, but not /path/to/program with other arguments.
 Cmnd_Alias MYPROGRAM = /path/to/program --option  
 artbristol ALL = (root) /path/to/program
 artbristol ALL = (root) NOPASSWD: MYPROGRAM
 Defaults!MYPROGRAM !requiretty
 ------------------------------------------------------------------------------------------
 #This allows the source (the sending host) to specify the route, loosely or strictly, ignoring the routing tables of some or all of the routers
 #source-based routing should be disabled. 
 /sbin/sysctl -w net.ipv4.conf.all.accept_source_route=0 #drop packets with the SSR or LSR option set

 #Disabling the forwarding of packets should also be done in conjunction with the above when possible (disabling forwarding may interfere with virtualization)
 /sbin/sysctl -w net.ipv4.conf.all.forwarding=0
 /sbin/sysctl -w net.ipv6.conf.all.forwarding=0
 #These commands disable forwarding of all multicast packets on all interfaces
 /sbin/sysctl -w net.ipv4.conf.all.mc_forwarding=0
 /sbin/sysctl -w net.ipv6.conf.all.mc_forwarding=0

 #Accepting ICMP redirects has few legitimate uses. Disable the acceptance and sending of ICMP redirected packets unless specifically required
 /sbin/sysctl -w net.ipv4.conf.all.accept_redirects=0
 /sbin/sysctl -w net.ipv6.conf.all.accept_redirects=0
 #disables acceptance of secure ICMP redirected packets on all interfaces
 /sbin/sysctl -w net.ipv4.conf.all.secure_redirects=0
 # disables acceptance of all IPv4 ICMP redirected packets on all interfaces:
 /sbin/sysctl -w net.ipv4.conf.all.send_redirects=0
 #automatically disable sending of ICMP requests whenever you add a new interface
 /sbin/sysctl -w net.ipv4.conf.default.send_redirects=0

 #make these settings persistent across reboots, modify the /etc/sysctl.conf file
 #disable acceptance of all IPv4 ICMP redirected packets on all interfaces, 
 #open the /etc/sysctl.conf file with an editor running as the root user and add a line as follow
 net.ipv4.conf.all.send_redirects=0
 ------------------------------------------------------------------------------------------
 #Reverse Path Forwarding is used to prevent packets that arrived through one interface from leaving through a different interface
 #unless you know that it is required, it is best enabled as it prevents users spoofing IP addresses from local subnets and reduces the opportunity for DDoS attacks. 
 #permanent changes can be made by adding lines to the /etc/sysctl.conf
 # make a temporary global change
 sysctl -w  net.ipv4.conf.default.rp_filter=integer
 sysctl -w net.ipv4.conf.all.rp_filter=integer

 ------------------------------------------------------------------------------------------
 #Enabling Packet Forwarding
 #enable packets arriving from outside of a system to be forwarded to another external host
 #change the line which reads net.ipv4.ip_forward = 0 in the /etc/sysctl.conf file to the following
 net.ipv4.ip_forward = 1
 /sbin/sysctl -p #load the changes from the /etc/sysctl.conf file
 /sbin/sysctl net.ipv4.ip_forward #check if IP forwarding is turned on,If it returns 1, then IP forwarding is enabled
 /sbin/sysctl -w net.ipv4.ip_forward=1 #If it returns 0, turn it on manually
 ------------------------------------------------------------------------------------------
 #check public IP,private (viewable within an internal network) or public (can be seen by other machines on the Internet)

 #3rd party web-sites
 $ wget -qO- http://ipecho.net/plain | xargs echo
 $ curl ifconfig.co
 $ curl ifconfig.me
 $ curl icanhazip.com
 $ curl -4 icanhazip.com
 $ curl -6 icanhazip.com
 $ curl ident.me
 $ curl checkip.dyndns.org
 $ curl api.ipify.org
 $ curl ipinfo.io/ip
 $ curl checkip.amazonaws.com
 ------------------------------------------------------------------------------------------
 #The Internet Assigned Numbers Authority (IANA) has assigned several address ranges to be used by private networks

    Class A: 10.0.0.0 to 10.255.255.255 
    Class B: 172.16.0.0 to 172.31.255.255
    Class C: 192.168.0.0 to 192.168.255.255
    
    Class A: 10.0.0.0/8 (255.0.0.0)	
    Class B: 172.16.0.0/12 (255.240.0.0)
    Class C: 192.168.0.0/16 (255.255.0.0)


 ------------------------------------------------------------------------------------------
 #Running GUI applications as root
 sudo /usr/bin/etherape
 ------------------------------------------------------------------------------------------
 chage -l ubuntu #last password change date
 ------------------------------------------------------------------------------------------
 #read first, and if a match is found the connection is allowed and the search is stopped.
 #If no allowed match if found, the hosts.deny file is read
 /etc/hosts.allow
 #If a match is found the connection is refused - otherwise it is allowed
 #/etc/hosts.deny
 ------------------------------------------------------------------------------------------
 # print server's IP on the wellcome page
 echo $(ifconfig eth0 | grep 'inet addr' | awk -F: '{ print $2 }' | awk '{ print $1 }') >> /var/www/html/index.html
 ------------------------------------------------------------------------------------------
 getfacl -a a.txt #file access control list of a file or directory.
 getfacl -t a.txt
 getfacl -n file #numeric user and group IDs
 getfacl -d a.txt #the default access control list of a file or directory.
 getfacl -R directory # the ACLs of all files and directories recursively (sub-directories)
 getfacl -L -R directory #follow symbolic links to directories. The default behavior is to follow symbolic link arguments and skip symbolic links encountered in subdirectories
 getfacl -P -R directory #do not follow symbolic links to directories,skip symbolic link arguments

 setfacl -m u:deepak:rw a.txt #grant read and write permission
 setfacl -b a.txt #remove all extended ACL entries,remove all entries
 setfacl -x u:deepak a.txt #remove user
 setfacl -x g:linux file #remove group
 setfacl -m g:linux:rw -R directory #remove group  recursively (sub-directories)
 setfacl -k file #remove the default access control list
 setfacl --test -x g:linux -R dir1 #The ACLs are not modified in test mode. It only displays the changes that will take place
 setfacl -dm "user:my_user:r--" /path/to/directory #Add a default entry to grant access to the user my_user on all newly created files within a directory


 getfacl file1 | setfacl --set-file=- file2 #copy the ACL of one file to another
 ------------------------------------------------------------------------------------------
 for i in *linux*; do rm $i; done #delete all the files in the current directory that contains the word “linux”
 cat linux.txt | grep n #list the entries that start has the character ‘n’
 cat linux.txt | grep ^a #ist the entries that start with the character ‘a’
 echo "shutdown now" | at -m 18:00 #shut down the system at 6 pm today
 vim -R <filename> #open a file in read-only mode 
 vim +/<employee id to be searched> <filename> #search for a specific Employee ID in a file
 ------------------------------------------------------------------------------------------
 cd $mydir && python3 -m http.server 8888 #Share a file quickly using a web server

 podman run --rm -v .:/usr/share/nginx/html:ro,Z -p 30080:80 -d nginx # Expose a web directory using containers
 ------------------------------------------------------------------------------------------
 $ sudo shutdown --halt 22:00 #--halt option stops the operating system
 $ sudo shutdown --halt +5 # halt the system after a five-minute delay
 $ sudo shutdown --halt +5 “Attention. The system is going down in five minutes.”
 $ sudo shutdown -c #Cancel a timed shutdown
 $ sudo shutdown -r now #restarting the system 
 $ sudo systemctl reboot #restarting the system 
 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 #how to delete file with inode
 find . -inum 1847 -ls
 find . -inum 1847 -exec rm {} \;

 find . -inum 782263 -exec rm -i {} \;

 #how to delete directory/folder with inode
 find . -inum 393232 -delete
 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 #The error message you're encountering seems to indicate a problem with the execution or syntax within the .bashrc file. The .bashrc file is a script that runs whenever a new terminal session is started for a user in Bash

 nano ~/.bashrc

 If there's no obvious syntax error, try validating the syntax of the .bashrc file using the bash command
 bash -n ~/.bashrc

 After making changes to the .bashrc file, restart the Bash shell or source the file to apply the changes

 source ~/.bashrc
 or
 exec bash
 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 #Listing a Directory with Spaces
 ls "My Documents"

 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------