Skip to content

Instantly share code, notes, and snippets.

@madaboutcode

madaboutcode/commandline-fu.md

Last active November 23, 2020 06:44
Show Gist options
  • Save madaboutcode/004ed09d1143f205441c to your computer and use it in GitHub Desktop.
Save madaboutcode/004ed09d1143f205441c to your computer and use it in GitHub Desktop.
Download ZIP
Command Line Magic™
Raw
commandline-fu.md

Command Line Magic

  • sort -u -t, -k1,1 file - Unique in column
    • u for unique
    • t, so comma is the delimiter
    • k1,1 for the key field 1
  • sed '/^$/d' myFile - remove blank lines from a file
  • find ~/ -type d -maxdepth 3 -exec du -hs '{}' \; - display usage for all folders in a directory
  • curl http://requestb.in/10wkj9v1 -x http://183.220.199.76:8123 - proxy testing
  • curl -o /dev/null -s -w "Connect: %{time_connect} \nTime to first byte: %{time_starttransfer}\nTotal time: %{time_total} \n" <url> - Webpage perf stats using curl
  • Portscan using netcat - nc -z -v 192.168.15.196 1-1000
  • non-root user to open privileged ports - setcap 'cap_net_bind_service=+ep' /path/to/program
  • move directory/files from a text file - while IFS= read -r file; do mv "$file" ~/newfolder; done < ~/Desktop/files.txt
  • rsync copy to remote over ssh - rsync -az ~/local/path user@host:/remote/path
  • give user permission to folder - sudo setfacl -R -m user:<username>:rwx folder/path
  • wget archive site - wget -mpck --user-agent="" -e robots=off --wait 1 http://blah
  • split json file into lines - sed $'s/\},/\}\\\n/g' rajib.json
  • add existing user to group - sudo usermod -a -G groupName userName
  • kill mutliple processed in one go - sudo ps -eopid,cmd | grep storm | cut -d ' ' -f1 | xargs sudo kill -9
  • replace text in all java files rg -tjava -l "AlertSetting" | xargs -n1 sed -i .bak 's/AlertSetting/AlertSettingNew/g'
  • jq - escape string - pbpaste | jq -c . | jq -R .

rsync

  • rsync as root but use localuser config - sudo rsync -e "sudo -u localuser ssh" -avz source destination
  • rsync as root on destination, no copy permissions - rsync -aPe ssh --rsync-path='sudo rsync' --no-p --no-o --no-g --delete src dest
  • exclude a particular file type: rsync -avz --exclude '*.txt' source/ destination/
  • exclude a folder rsync -avz --exclude 'dir1' source/ destination/

grep

  • grep -nr yourString* . - grep search for a string in a directory recursively
  • grep -lr '<string to find>' /var/cache/nginx/<folder>* | xargs rm: nginx cache delete
  • show before and after few lines grep -B 3 -A 2 foo README.txt

find

  • find with max-depth 2, modified in last 20 days - find . -mtime -20 -maxdepth 2 -exec ls -ld "{}" \;

Postgres

  • setup new database for dev
    • sudo su - postgres
    • createdb <dbname>
    • createuser <username>
    • psql <dbname>
    • alter user <username> with encrypted password '<password>'
    • grant all on database <dbname> to <username>

Awk

  • awk '/start_pattern/,/stop_pattern/' file.txt - Display a block of text with AWK

Tor

  • torify <command> - routes all network connections through tor for a particular command

FFMPEG

  • ffmpeg -i 00079.MTS -c:v libx264 -preset veryslow -crf 23 -maxrate 6000k -bufsize 2M -c:a libfdk_aac -b:a 128k 00079_slowest.mp4 - lossy, but 1080p HD quality video
    • reduce the bufsize for preventing lag during seeking.
  • ffmpeg -i VID_20141229_212438713.mp4 -metadata:s:v rotate="0" -vf "transpose=1" -c:v libx264 -preset slow -crf 18 -maxrate 6000k -bufsize 500K -c:a libfaac -b:a 128k VID_20141229_212438713_o.mp4
    • With rotate and lower quality
  • ffmpeg -i myvideo.avi -f image2 -vf fps=fps=1/60 img%03d.jpg - generate thumbnails every 60 sec from a video.
  • ffmpeg -s 1920x1080 -f v4l2 -vcodec h264 -i /dev/video0 -f pulse -i "alsa_input.usb-046d_HD_Pro_Webcam_C920_55571E2F-02-C920.analog-stereo" -acodec libfdk_aac -ac 1 -b:a 128k -copyinkf -vcodec copy capture-$(date +"%Y%m%d_%H%M%S").mp4.mp4 - record webcam audio video
  • ffmpeg -r 1 -loop 1 -t 61 -i Screenshot_2019-12-06\ MPU-6050\ TDK.png -filter_complex "color=white:s=1280x720, fps=fps=60[bg];[bg][0]overlay=y=-'t*120':shortest=1[video]" -preset ultrafast -map "[video]" output.mp4 - create a smooth vertical scrolling video of an image file
  • ffmpeg -i input.mp4 -vf "rotate=rotate=PI:bilinear=0,format=yuv420p" \-metadata:s:v rotate=0 -codec:v libx264 -codec:a copy output.mp4 - rotate 180deg

VLC

-cvlc v4l2:///dev/video0:chroma=h264:width=1920:height=1080 --sout '#standard{access=http,mux=ts,dst=localhost:8080,name=stream,mime=video/ts}' -vvv streaming from webcam

OSX

  • osascript -e 'display notification "Did you log your time?" with title "Ding Ding!"' - use AppleScript to show notification
  • mdfind recipes -onlyin ~/Documents/ - spotlight from command line
  • sudo networksetup -setdnsservers Wi-Fi 127.0.0.1 - set dns from commandline; reset it use sudo networksetup -setdnsservers Wi-Fi empty

Git

  • files list for ajeesh for the past 5 days - git log --name-only --since="5 days ago" --author=ajeesh.p.mohan --pretty=format: | sort | uniq
  • delete last local commit that has not been pushed - git reset --hard HEAD^
  • show commits with deleted files - git log --diff-filter=D --summary
  • If you need to generate a git log of all your commits to all branches in all git repos under the current dir fd -d 1 -t d -x git -C {} log --pretty=format:"%ad:{}:%an:%d:%s" --date=short --reverse --branches --since=2.months.ago --author=Ajeesh | sort
  • git branch cleanup report -
for branch in `git branch -r | grep -v HEAD`;do echo -e `git show --format="%cI|%an|%cr|" $branch | head -n 1`"$branch|"$(git-is-merged origin/2.4 $branch) ; done | sort -r
  • git-is-merged :
 #!/bin/bash

merge_destination_branch=$1
merge_source_branch=$2

merge_base=$(git merge-base $merge_destination_branch $merge_source_branch)
merge_source_current_commit=$(git rev-parse $merge_source_branch)
if [[ $merge_base = $merge_source_current_commit ]]
then
	echo MERGED
	exit 0
else
	echo UNMERGED
	exit 1
fi

iptables

  • sudo iptables -L --line-numbers - List existing rules with linenumbers
  • sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT - Add rule to accept port 22
  • sudo iptables -I INPUT 1 -i lo -j ACCEPT - Insert rule a position 1 to accept everything on loopback interface
  • sudo iptables -A INPUT -j DROP - Add a rule to drop all packets. Place this at the bottom of the ruleset
  • sudo iptables-save - persist the rules. need sudo apt-get install iptables-persistent

attic

  • attic list ~/backups/projects.attic - list all version for the archive
  • attic list ~/backups/projects.attic::2015.03.13_12.10AM - list files in an archive.
  • attic extract /path/archive.attic::backupname "full_path_from_root" --strip-components 15 - extract a file from attic,
    • --strip-components Remove the specified number of leading path elements. Pathnames with fewer elements will be silently skipped.

vim

  • :w !sudo tee % - write as sudo

SAR

  • sadf -dh -- -r /var/log/sa/sa08 - memory usage for 8th of the month in csv format
  • sadf -dh -- /var/log/sa/sa08 - cpu usage for 8th of the month is csv format

siege

  • siege -c 300 -t 10m -b -q 'http://example.com POST <data.json' - POST Data concurrency-300, time-10m, -b benchmark - no delay

zmv - zsh move command

  • autoload -U zmv - load zmv - do this first
  • zmv '(*).txt' '$1.html' - as you can guess
  • zmv -n .... - -n to dryrun first

socat

Hosts: a server, blocked by a firewall a client outside the firewall ("outside-host") a firewall that allows arbitrary TCP connections from server to outside, port 80

  1. Start the double client on the inside server // every 10 seconds, it tries to establish a connection to the outside host. // whenever it succeeds, it forks a sub process that connect to the internal // service and starts to transfer data $ socat -d -d -d -t5 tcp:outside-host:80,forever,intervall=10,fork tcp:localhost:80

  2. Start double server on the outside client // wait for a connection from a local client. whenever it accepted it, forks // a subprocess that tries to bind to the socket where the inside double // client tries to connect (might need to wait for a previous process to // release the port) # socat -d -d -d tcp-l:80,reuseaddr,bind=127.0.0.1,fork tcp-l:80,bind=outside-host,reuseaddr,retry=10

nmap

  • sudo nmap -sS -p 20-25 127.0.0.1 - TCP SYN Scan for open ports

Borg

  • borg init -e "keyfile-blake2" --append-only user@host:/backup/borg
  • borg key export user@host:/backup/borg keyfile.txt

mmv

  • mmv '*Episode [1-9]-*v.*' '#1Episode 0#2-#3v.#4'

caddy

  • https://caddyserver.com/download/linux/amd64?plugins=http.cache,http.cors,http.expires,http.forwardproxy,http.login,tls.dns.cloudflare&license=personal&telemetry=off

Bash Scripting

File exists

if [ -f $FILE ];
then
   echo "File $FILE exists."
else
   echo "File $FILE does not exist."
fi

loop and upload

find . -type f -name "*.jpg" | while read f 
do
    curl --upload-file $f "https://transfer.sh/`basename $f`"
done

if return code

if [ $? -eq 0 ]; then

fi

string comparison

if [ "$x" = "valid" ]; then
  echo "x has the value 'valid'"
fi

if [ "$x" != "valid" ]; then
  echo "x has the value 'valid'"
fi

if directory does not exist

if [ ! -d "$REPOSITORY" ]; then
    exit 1
fi

scripts

monitor if internet is up in dlink modem

while true;do
  curl --user "user:password" --silent -q 192.168.1.1/wancfg.cmd?action=view | grep -i "<td>Up</td>" > /dev/null && printf "\b\b\b\bUP" || printf "\b\b\b\bDOWN"; 
done

#reference/commands

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment