Command line tools every developer should know

gistfile1.md

Command line tools every developer should know

Prior reading

• The Unix philosophy, specially the "Make each program do one thing well" [1]

Basic

• File and directory navigation/manipulation (ls, cd, mkdir, rm, rmdir, touch, cp, mv)
• ln/unlink/readlink
• find/locate
• chmod
• chown
• man/info
• su/sudo
• set/export (environment variables in general, specially PATH)
• which
• df
• history

Text editing

• vi (enough to open/edit/save a file)

Text processing

• cat
• head/tail
• more/less
• grep
• awk/sed/cut
• (regular expression in general)
• wc
• diff
• tr

Streams, pipes and redirects

• <<
• |
• xargs
• tee

Networking

• ping
• netstat
• ssh/scp
• ngrep
• netcat
• curl
• wget
• telnet (mostly for port testing?)
• rsync
• lsof (-i)

Processes

• ps
• uptime
• top/htop
• nohup
• kill/pkill/killall
• &
• (Ctrl + Z)
• fg/bg/jobs
• crontab
• lsof

File archiving and compressing

• tar (including manipulating gzip/bzip)
• zip

References

[1] The Art of Unix Programming (http://www.catb.org/esr/writings/taoup/html/ch01s06.html)

Resources

• Manual pages ('man' command)
• SS64 command references: http://ss64.com/bash/
• Useful command snippets: http://commandlinefu.com
• Useful aliases: http://alias.sh
• Command Line Crash Course: http://cli.learncodethehardway.org/book/

I'm assuming people will have to learn the concepts required for using each command. But you're right, some commands do require more background, like permissions, streams and processes.

Also agreed that nowadays people mostly use tar/zip. I'll change that and also add htop/sudo/su.

Thanks!

Maybe cp and mv (as well as some of the options like -r).

Use of adding: & to end of a command or ctrl+z (I think?) then: bg

Use of "which"

If they're beginning development for X language, they might need to know how to get to the configuration file. I know I always have to edit the php.ini file, so this is useful for me on machines that I'm not familiar with: php --ini

Also: git, svn? rsync (maybe?)

I think you've got most of the basics. To check I ran history on my oldest current terminal and you've captured 95% of my command line usage. I would also suggest process control, (bash specific) Ctrl-Z to suspend, bg to background, fg to foreground. vim, change code, Ctrl-Z, run tests, fg, change code is a frequent pattern for me. Also, no love for less?

Make, or the build tool most developers use for your language of choice, so you better understand managing dependencies among the files in your code base, and the various possibilities of your build tool chain.
Rpm, jar, or the packaging tool most devs... , so you understand what you can deliver to your customer, and what her installation looks like.
In general, reduce reliance on your IDE, so you're more portable.

great list! re tail, make sure that tail -f gets an explicit mention...

@dan2k3k4 Well spotted. I'll add cp/mv/which to the list. Maybe the basics about environment properties would also be useful (specially to solve common PATH issues). I'll probably leave tools that are closer to development toolchain (like git) and service-specific (like php.ini) out of this first draft until I have a good idea of where this list is going.

@aychedee (and also @dan2k3k4) You're right. I can probably expand on process control, as it's pretty common stuff. Not the most basic, but definitely useful.

@hjwp Added. Thanks.

Added wc/diff thanks to @lamsap's suggestions.

Added suggestions around unix philosophy and xargs (Thanks, @SamirTalwar)

Networking:
ifconfig, netstat (Unix)
ipconfig, net * (Windows)

I would add dd for delete line to key vi commands.

"vi is a definite, even if it's just the very basics (i, ESC, w, q, dd)."

I find cut very useful - a lot simpler than awk or sed for extracting values from structured text

@professor Good point. "i", ESC, dd (and maybe "p") are the basics of editing in vi.

@alxndrsn I'll add your suggestion. Thank you.

Added another resource, the Command Line Crash Course from Zed Shaw. (Thanks, @sit)

Good list!

Might it be worth splitting it, or each section, into "fundamentals" and "more" sections? Everyone should know cd and ls, but many people will be able to get by without ln and find.

Specific tweaks i would suggest:

rmdir - delete a directory, but only if it's empty; this can be safer than using rm -r, because you won't accidentally delete a non-empty directory
chgrp - i wouldn't mention this, as you can and usually do change the group with chown at the same time as changing the owner (chown sanchez:users)
info - most unix tools have man pages, but quite a lot of GNU tools have very small man pages with more documentation in an info manual; the man page usually says so, though, so this may not be important to know upfront
local variable exports - saying SOMEVAR=foo bar will run the bar command in an environment where SOMEVAR is exported with the value foo; this is often a more concise and hygienic way of exporting variables than with an explicit export
which - use the -a flag (on Linux, and i think also OS X / FreeBSD, i think) to find all the commands with a given name, not just the first one
readlink - get the destination of a symlink; readlink -f (GNU readlink only, ie Linux) fully resolves all the symlinks in a path and prints an absolute, physical, path, which can be incredibly useful
watch - this is crazy useful
find - very useful for searching for files, with lots and lots and lots of flags to filter things out
locate - if the locate database is installed (it is on most Linux desktops, often not on servers), quickly search for a file anywhere on disk by name

tr - translate one character to another; mostly useful for adding or removing line breaks in pipelines
head - handy to know that you can use negative offsets to print all but the last N lines of a file
tail - handy to know that you can use an offset with a plus to print all but the first N lines of a file (eg tail -n +2 skips the first line)
tail -f - PLEASE DO NOT TELL PEOPLE TO USE tail -f! In a world where less +F exists, it is completely obsolete. less +F is miles better, and it drives me bonkers to see people mucking about with tail -f.
less - rewards study; useful to know how to search (/) and filter (&), as well as to follow (F, control-C to stop following) - in particular, the combination of filtering and following is a great way to sieve useful information from a busy log
grep - there are so many useful options to grep that it merits an article of its own. -o is a personal favourite which can often replace a use of sed
awk - i know awk, and i like it, but i barely ever find a need for it; a good knowledge of sed is much more usefl
sed - sed is wonderful, although completely impenetrable beyond simple uses; it's the assembly language of text processing
cut - highly underrated, a go-to tool in many of my pipelines
comm - similar to diff, compares two files and prints lines unique to one or the other or present in both; generally more useful in scripts and pipelines than diff
cmp - a bit like a baby diff, really only useful in tests (eg if cmp referencefile $MYFILE; then ...)

$() - command substitution, a wonderful tool
<() and >() - process substitution, obscure but at times hugely useful
tee - copy a stream into a file (or a process substitution!) and also pass it on along a pipeline
<< - handy to know the variant forms <<- which strips leading whitespace, letting you indent the here-doc nicely in a script, and <<'EOF', which prevents variable expansion in the here-doc
xargs - worth knowing about -n 1 (only execute for one parameter at a time, useful with commands that only take a single argument) and -0 (break arguments on nulls not whitespace - useful if arguments have whitespace in, which can be produced with the -print0 flag to find for files, and with a suitable tr stage in a pipeline for other sources)
while and for - handy to know about the use of the shell's looping constructs as an alternative to xargs; more verbose but more flexible

ping - another useful network tool!
lsof - with suitable flags, useful for seeing what sockets a process has open, and often more reliable for this than netstat
netstat - still convenient
wget - similar to curl, mostly not as flexible, but has one really useful feature, namely the ability to read a list of URLs from a file (or stream) and download them all, making it very useful as the terminus of a pipeline

pgrep - basically an all-in-one ps -ef | grep, but more accurate, very usefl
pkill - a pgrep that kills anything it finds; means you can say "pkill java" to kill java
killall - a slightly safer pkill
lsof - i mention this again here because it's also incredibly useful for seeing what files a process has open - a classic use of this is seeing where a program is writing its f*****g log file
jobs - tells you what you have running in the background (via & or bg)

df - see how much free space there is on the disk
mount -l - see what filesystems are mounted where

package manager commands - depends on the package manager, but it's useful to know how to search for a package, install it, remove it, list the contents of a package, and find which package owns a file (the latter is a vital trick for the kind of detective work i often find myself doing)

I'll let you know if i think of any more!

Great additions, Tom - never heard of watch, and didn't know about less's follow and filter options.

+1 for tee as well

@tomwhoiscontrary Thanks A LOT for the contributions. It will take me a bit to go through all of them...

But as a start I removed the "-f" reference. I was not aware of the less option. It does seem like a better option indeed.

I'm overwhelmed by the feedback I've got so far. Thank you all for the help.

Seriously considering moving this into something a bit more well structured than a gist...

I'd switch from "vi" to something along the lines of "vi, emacs, nano, or another ubiquitous term based text editor"

@jredville the reason I only mention vi is because I'm yet to see a unix box where it's not there. The same does not apply to other editors, so that's why I believe it's essential to know the minimum of vi in case you get to a machine without install privileges.

Added more commands (Thanks again, @tomwhoiscontrary).

tmux - I think you should add this as well. It's so very handy when working on multiple servers and multiple tasks.