Skip to content

Instantly share code, notes, and snippets.

@jamesyang124

jamesyang124/cmd_tools.md

Last active February 28, 2020 17:38
Show Gist options
  • Save jamesyang124/994cc7e22e7a81b31cda to your computer and use it in GitHub Desktop.
Save jamesyang124/994cc7e22e7a81b31cda to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cmd_tools.md

Content Search

  • ack/ag, examples:
ag strings [path|files] --ignore="*.yml" --ignore="*.js.coffee"

# search readme at end of line
ag readme$
  • Both Ack and Ag treat the search term as a regular expression by default. -Q for exactly pattern.
# may only produce single char [.]
ag .rb railties/CHANGELOG.md

# use -Q to search by exactly pattern
# not regex
ag -Q .rb railties/CHANGELOG.md
  • -l to list file names only, not dsiplay line numbers.
ag DHH -l
  • -i case sensetive.
ag readme -l -i
  • We can also scoping for the specific file name patterns by -G, the patterns also treat as regexp.
ag readme -l -G action
# railties/lib/rails/generators/actions.rb
# railties/test/generators/actions_test.rb

ag readme -l -G ec$
# actionmailer/actionmailer.gemspec
# actionpack/actionpack.gemspec
# actionview/actionview.gemspec
  • ignore by dir: --ignore-dir
ag readme -l --ignore-dir=railties/lib --ignore-dir=guides/code

ag readme -l --ignore-dir="*.rb"
  • ag supports VCS(version control system) Ignore Files. So file list in .gitignore will also be ignored by ag in its seraching. To turn off this feature, use --skip-vcs-ignores.
ag readme -l --skip-vcs-ignores

  • Or other way to do so, use .agignore instead.

  • use ag to search from STDOUT

ps -e | ag forego

Find File by Query Terms

find PATH_TO_SEARCH OPTIONS_TO_USE PATTERN_TO_SEARCH_FOR
  • by name -name or -path.
find . -name model.rb
# active_model/model.rb
# active_record/model/templates/model.rb

# use \ to escape for wild char *
find . -name \*model.rb
# ./activemodel/lib/active_model.rb
# ./active_record/model/templates/model.rb
# ./models/arunit2_model.r

find . -path \*session\*
  • We can provide sub command for finding only files or dirs by -type [f|d]
# files only
find . -path \*session\* -type f

# dirs only
find . -path \*session\* -type d
  • Conjunct two queries by and or or, by default is and operation.
find . -path \*session\* -type f -name \*mem\*
  • We can also do OR queries in find using the -or operator and some well placed parentheses.
find . \( -name \*.gemspec -or -name \*.jpg \) -type f

# ./actionmailer.gemspec
# ./fixtures/attachments/foo.jpg
# ./fixtures/attachments/test.jpg
  • Not matching query ! or -not.
find . -not -path \*t\* -type f

find . \! -path \*t\* -type f
  • Find by last modified time -mtime, size -size, last modifed mins -mmin
# one days ago -1
find . -mtime -1

# 5 mins ago
find . -mmin -5

# by size, 
# +k = Kb, M = Mb, G = Gb,
# T = Terab, P = Petab 
find . -size +200k
  • performing operations such as delete and print once found it.
find ./guides -type f -name \*.yml -print -delete

Monitoring Process

  • ps, common cloumns:
  • tty to get current terminal id.
# PID process ID
# TTY Terminal ID
# TIME cumulative time since process started
# CMD command name
  • display user oriented output by u
ps u

#VSZ	Indicates, as a decimal integer, the size in kilobytes of the process in virtual memory.
#RSS	The real-memory size of the process (in 1 KB units).
#STAT	The status of the process.
#STARTED	When the process was started.

  • display all process e or ax, slightly different columns.

  • -U flag will filter the results to show only processes that are being run by the user ID provided.

ps -U root
  • Customize displayed columns by -O, list all options by -L.
ps -L
# %cpu %mem acflag acflg args blocked caught comm 
# command cpu cputime etime f flags gid group ignored 
# inblk inblock jobc ktrace ktracep lim
# login logname lstart majflt minflt msgrcv msgsnd ni 
# nice nivcsw nsignals nsigs nswap nvcsw nwchan oublk 
# oublock p_ru paddr pagein pcpu pending
# pgid pid pmem ppid pri pstime putime re rgid rgroup 
# rss ruid ruser sess sig sigmask sl start stat state 
# stime svgid svuid tdev time tpgid
# tsess tsiz tt tty ucomm uid upr user usrpri utime 
# vsize vsz wchan wq wqb wql wqr xstat

ps -O %cpu,command,etime
  • sort by memory -m or CPU usage -r. we’ll use the -u flag to scope results to the root user.
ps -m -O %mem -u root
ps -r -O %cpu -u root
  • ps -L
Column Description
%cpu percentage CPU usage (alias pcpu)
%mem percentage memory usage (alias pmem)
acflag accounting flag (alias acflg)
args command and arguments
comm command
command command and arguments
cpu short-term CPU usage factor (for scheduling)
etime elapsed running time
flags the process flags, in hexadecimal (alias f)
gid processes group id (alias group)
inblk total blocks read (alias inblock)
jobc job control count
ktrace tracing flags
ktracep tracing vnode
lim memoryuse limit
logname login name of user who started the session
lstart time started
majflt total page faults
minflt total page reclaims
msgrcv total messages received (reads from pipes/sockets)
msgsnd total messages sent (writes on pipes/sockets)
nice nice value (alias ni)
nivcsw total involuntary context switches
nsigs total signals taken (alias nsignals)
nswap total swaps in/out
nvcsw total voluntary context switches
nwchan wait channel (as an address)
oublk total blocks written (alias oublock)
p_ru resource usage (valid only for zombie)
paddr swap address
pagein pageins (same as majflt)
pgid process group number
pid process ID
ppid parent process ID
pri scheduling priority
re core residency time (in seconds; 127 = infinity)
rgid real group ID
rss resident set size
ruid real user ID
ruser user name (from ruid)
sess session ID
sig pending signals (alias pending)
sigmask blocked signals (alias blocked)
sl sleep time (in seconds; 127 = infinity)
start time started
state symbolic process state (alias stat)
svgid saved gid from a setgid executable
svuid saved UID from a setuid executable
tdev control terminal device number
time accumulated CPU time, user + system (alias cputime)
tpgid control terminal process group ID
tsess control terminal session ID
tsiz text size (in Kbytes)
tt control terminal name (two letter abbreviation)
tty full name of control terminal
ucomm name to be used for accounting
uid effective user ID
upr scheduling priority on return from system call (alias usrpri)
user user name (from UID)
utime user CPU time (alias putime)
vsz virtual size in Kbytes (alias vsize)
wchan wait channel (as a symbolic name)
wq total number of workqueue threads
wqb number of blocked workqueue threads
wqr number of running workqueue threads
wql workqueue limit status (C = constrained thread limit, T = total thread
xstat exit or stop status (valid only for stopped or zombie process)
Signal Definitions for ps -l

Tar

  • -c to archive files.
# archive all files under tools dir
tar -c tools/

# Unix meta-data
#tools/000755 000765 000024 00000000000 12244661361 013337
# 5ustar00markbatesstaff000000 000000 tools/console000755 000765 000024 00000000255
# 12244661361 014731 0ustar00markbatesstaff000000 000000 #!/usr/bin/env ruby

  • The apparent gibberish preceding each of the files that makes up the archive. This "gibberish" is, in fact, the Unix meta-data of that file; including such things as the permissions, file’s owner, group, etc - all valuable information to be retained.

  • The second thing that should be noted is that the archive was printed to "standard out" (STDOUT).

  • save to file by -f, verbose mode

tar -cfv foo.tar tools/

# multiple files
tar -cvf foo.tar tools/ tasks/ install.rb rails.gemspec version.rb
  • Using the -T flag and the pipe operator (|), we can use find to generate a list of files and then use tar to archive those files.
find . -name \*.gemspec | tar -cvf gemspecs.tar -T -

  • The -T flag will read in a list of file names from another file. In our case we are going to use - in place of the name of a file. This tells tar that it should read in the list of files from the find query we piped in.

  • list files in an archives by -t.

tar -tf foo.tar

#tools/
#tools/console
#tools/profile
#tasks/
#tasks/release.rb
#install.rb
#rails.gemspec
#version.rb
  • appending files by -r
tar -rvf foo.tar Gemfile README.md
  • update files that are already in the archive by -u, NOT COMPATIBLE IN COMPRESSED ARCHIVES.
tar -uvf foo.tar Gemfile README.md
  • extract file by -x
tar -xvf foo.tar

# extract single file
tar -xvf foo.tar rails.gemspec

# extract to different dirs
tar -xvf foo.tar -C ./myfolder
  • Compressing with gzip -z, bzip2 -j, Unix compress -C. bzip2 is considerably slower for compression and decompression, but produces smaller archives, and Unix compress is faster, but produces considerably larger archives.
# just create archive uncompressed
tar -cf ar.tar activerecord/

tar -czf ar.tar.gz activerecord/
# -rw-r--r--  1 markbates  staff  3311104 Nov 26 14:12 ar.tar
# -rw-r--r--  1 markbates  staff   615262 Nov 26 14:14 ar.tar.gz
  • Extracting compressed files by -z
tar -xvzf ar.tar.gz

Grep

  • Similar function as ag/ack.

  • Regexp in search pattern by default:

grep "r..y" README.md
  • counting occurence by -c
grep -c ruby README>md
  • display line numbers -n, case sensitive in -i
grep -n ruby README.md
grep -n ruby README.md RELEASING_RAILS.doc *.rb
  • recursive searching by -R, use to search all fles in dirs.
grep -R "Read" . config/
  • also we can specify which files pattern by --include=
grep -R --include="*.yml" "Read" . config/
  • invert searches by -v to find files which does not contain seach pattern.
grep -v "a" README.md

Curl

  • use -i to respond more informations, including headers, server type, content type, and more.

  • use -L to following redirection request.

curl -L quiet-waters-1228.herokuapp.com/redirectme
  • use -O to download files with default file name, use -o to custom its file name.
curl -o my_image.jpg http://quiet-waters-1228.herokuapp.com/assets/image.jpg

curl -O http://quiet-waters-1228.herokuapp.com/assets/image.jpg
  • Change HTTP request method by -X
curl -X POST quiet-waters-1228.herokuapp.com/echo
curl -X PUT quiet-waters-1228.herokuapp.com/echo
curl -X OPTION quiet-waters-1228.herokuapp.com/echo
  • Sending query string, or data payload in reuqest body by -d.
# by query string
curl -X POST "quiet-waters-1228.herokuapp.com/echo?fname=Mark&lname=Bates"

# by request body
curl -X POST -d "fname=Mark&lname=Bates" quiet-waters-1228.herokuapp.com/echo

# send json text
curl -X POST -d "{\"name\":\"Mark\"}" quiet-waters-1228.herokuapp.com/echo

# read file as data, prefix @
curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo
  • or passing form data in request body, then we can also upload files by @ for specific form field.
curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \
  quiet-waters-1228.herokuapp.com/echo -H "Accept: application/json"
  
curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \
  -F user[photo]=@path/to/image.jpg quiet-waters-1228.herokuapp.com/echo
  -H "Accept: application/json"
  • Setting header by -H
curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo \
  -H "Accept: application/json" -H "X-Auth: 1234567890"
  • Basic HTTP Auth -u
curl -X POST -u "user1:password1" quiet-waters-1228.herokuapp.com/login
  • In authentication, you might need to store and send cookies by -D(--dump-header) to dump it to a named fileds and -b respectively.
# dump header to headers file
curl -X POST -D headers -u "user1:password1" \
  quiet-waters-1228.herokuapp.com/login

# send headers file as header back
curl -X POST -b headers -u "user1:password1" \
  quiet-waters-1228.herokuapp.com/login

  • When using the –dump-header flag we not only capture the cookies from a response, but also all of the headers from that response.

  • Using cookie Jar by -c, curl has a built-in cookie jar meant for storing just the cookies and sending them back to the server.

url -X POST -c cookies.txt -u "user1:password1" \
  quiet-waters-1228.herokuapp.com/login

cat cookies.txt

# send back by b
curl -b cookies.txt quiet-waters-1228.herokuapp.com/whoami

Sed

  • sed is an editor that allows us to use pattern matching to search and replace within a file, or an input stream. This works by using Regular Expressions.

  • By default, the results of any edits we make to a source file, or input stream, will be sent to STDOUT.

  • The first argument is what we want sed to do, in the form of s/regexp/replacement/. It’s important to note that I wrapped the first argument in quotes to escape it.

sed "s/[aeiou]/*/" birthday.txt

# H*ppy Birthday to You, cha, cha, cha.
  • Replace the nth occurrence of a pattern.
# replace 2nd substrs.
sed "s/cha/foo/2" birthday.txt
  • save files, inplaced replacement by -i or provide file name
sed -i "s/(name)/Mark/" birthday.txt

sed -i.tmp "s/(name)/Mark/" birthday.txt
# ls
# ... birthday.txt
# ... birthday.txt.tmp
  • Case insensetive searches s/match/replacement/i.
sed "s/happy/Merry/i" birthday.txt
  • Matching line by lines. /g
sed "/cha/s/./*/g" birthday.txt
  • change case by either \u or \l.
sed 's/[aeiou]/\u&/g' birthday.txt

sed 's/[A-Z]/\l&/g' birthday.txt
  • Controlling output, -n suppress the output to STDOUT.
sed -n "s/(name)/Mark/" birthday.txt
  • only print the modified content by /p.
 sed -n "s/(name)/Mark/p" birthday.txt
  • Writing modified content to a File /w, combine with -n and /p.
sed -n "s/(name)/Mark/pw birthday2.txt" birthday.txt
  • displaying specific lines:
sed -n '1~2p' birthday.txt
# This argument can be read as starting at line 1 print (p) every 2nd line.
  • Delete lines, /d
$ sed "/(name)/d" birthday.txt
  • we can use pipe to chain the results.
#<p>
#  This <i>is</i> some <b>HTML</b>.
#</p>
#<a href="*.com">Conquering the Command Line</a>


# replace <[^>]*> tags to empty
sed 's/<[^>]*>//g' example.html
# blank sapce still left.
#  
# This is some HTML.
#  
#Conquering the Command Line


sed 's/<[^>]*>//g' example.html | sed '/^$/d'
  • Or running multiple sequence by ;
sed 's/<[^>]*>//g;/^$/d' example.html

Extras

cal

cal is a calendar cal -y shows current year, cal 2015 shows 2015 year, and cal 8 2015 shows month of year.

cat

The cat command is short for concatenate.

We can read multiple files such as cat file1 file2.

By the > redirection operator, output to the files ex: cat file1 file2 > outputfile.

cat -b big_file.txt to display line numbers.

kill

A few of these signals will forcefully terminate a process. Others will simply tell the process to shutdown when it is ready.

Signal Value Action Comment
SIGHUP 1 Term Hangup detected on controlling terminal or death of controlling process
SIGINT 2 Term Interrupt from keyboard
SIGQUIT 3 Core Quit from keyboard
SIGILL 4 Core Illegal Instruction
SIGABRT 6 Core Abort signal from abort(3)
SIGFPE 8 Core Floating point exception
SIGKILL 9 Term Kill signal
SIGSEGV 11 Core Invalid memory reference
SIGPIPE 13 Term Broken pipe: write to pipe with no readers
SIGALRM 14 Term Timer signal from alarm(2)
SIGTERM 15 Term Termination signal
SIGUSR1 30,10,16 Term User-defined signal 1
SIGUSR2 31,12,17 Term User-defined signal 2
SIGCHLD 20,17,18 Ign Child stopped or terminated
SIGCONT 19,18,25 Cont Continue if stopped
SIGSTOP 17,19,23 Stop Stop process
SIGTSTP 18,20,24 Stop Stop typed at tty
SIGTTIN 21,21,26 Stop tty input for background process
SIGTTOU 22,22,27 Stop tty output for background process

Default signal is 15, SIGTERM.

kill 123456

Sending a signal to process with a PID of 123456 to peacefully shutdown when it is ready.

Forcefully terminate a process.

kill -9 123456

pbcopy & pbpaste

MacOS only, use xsel of X11 tools instead in Linux.

pbcopy < birthday.txt
pbpaste > example.txt
pbpaste | ag name

tail

The tail command allows us to look at the end of a file.

We can list last n lines by -n, and also reverse output by -r(from bottom to top).

tail -n 5 example.logs

tail -r -n 5 example.logs

Watch a "LIVE" file by -f, When using the -f flag, tail will never exit and will continue to output new data as it is added to the end of the watched file.

tail -f /var/log/system.log
tail -f /var/log/system.log

Caveat!

When I save the file opened in vim, it creates a new file with a new inode while the one opened by tail is still the old one (which is now a temporary file which has actually been deleted).
http://tech.shantanugoel.com/2009/12/23/continuous-monitor-tail-fails.html

So use -F or --follow=filename instead.

the -F tells tail periodically reopens the file by name and reads it again, thus bypassing the above issue.

We can combine with ag to search output.

tail -f /var/log/system.log | ag com.apple.windowserver

tree

Limit depth of listing tree -L 2.

Listing dirs only by tree -d.

Listing another dirs, not current pwd, tree -d /usr/local/etc/.

wc

Word/line counters.

counting words.

wc -w /usr/share/dict/web2a

counting lines in a file.

wc -l /usr/share/dict/web2a

piping

| pipe the STDOUT to STDIN.

fswatch

To watch any changed files in a folder through watch(Linux) or fswatch(MacOs). Then we can chain the output to run scripts for it.

brew install fswatch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment