Unix Command Line Notes

Unix Command Line Notes.md

Unix Command Line Notes

• Command settings

  • .bashrc
    • Aliases - place the following on a new line in the .bashrc file
      • alias [alias-name]='[alias command(s)]
      • Ex:alias ls='ls -lh'
  • .profile or .bash_profile
    • Add folders to you path
    • Use source .profile to reload the profile
  • history
    • By default, remembers the previous 100 commands you've typed
    • history will list the stored comands
    • ![number] will repeat one of the commands storied in the history
    • history | grep [search] will display only the commands in the history that match your search query
  • man [command] to get documentation for a particular command
    • q will exit out of manual pages

• Manipulating files

  • pwd - print working directory
  • df - report on disk usage
  • ls /[directory-path] - list files within a directory
    • Specifying a directory is optional; will list the current directory by default
    • -a show hidden files and directories (all files)
    • -d gets directory information (in addition to the files in that directory)
    • -l long listing
    • -R list files recursively
    • -F classify files with special characters: display a / after each directory, @ after each link, and * after each executable (normal files have nothing special)
    • -s include the number of blocks the file takes up (for more accurate space indication of what the file actually uses on the disk)
    • -h for human readable file sizes
    • -e show ACLs, if present
    • -t shows files from most recently edited
    • −1 lists files in a single column
    • -r reverse the sort order
  • cp [filename/folder] [destination] - copies files
    • Automatically overwrites destination
    • -i asks for confirmation before overwriting existing files
    • -a preserve symbolic links (-d), recusively (-R), and reserve attributes (preserve=all)
  • rm [file/folder] - remove a file or directory
    • Normally, a directory must be empty to remove it
    • -f force removal
    • -r remove recursively (everything inside)
    • -i ask for confirmation
    • -d removes directories
    • -v for verbose output
  • mv [file] [new-file-name] or mv [file] [/new-destination] - renames or moves files or folders
  • du - space a file/directory takes up
  • touch [filename]
    • For existing files, updates the accessed time
    • If the file doesn't exist, it creates a new, blank file
  • cat [filename] - displays the contents of a file
  • Replace or add to files
    • "[contents]" > [filename] will create a new file or replace the existing file with the contents you specify
    • "[contents]" >> [filename] will append the contents to an existing file (or create a new file with those content)
  • Symlinks (for soft links, which can span partitions)
    • ln -s [/path/destination] [sym-link name]
  • rsync [options] [source] [destination]

    • Copies only the folders/files that have been changed from one location to another

    • Can be used over network connections (local <-> remote)

    • End the path with / to indicate copy just the contents of a folder, and not the folder itself

    • Remote destinations can be indicated through ssh user@server:/path/to/files

    • rsync to backup a Mac system volume

        sudo rsync -avPE --exclude "/Volumes" --exclude "/dev" --exclude ".MobileBackups.trash" / /destination
      

    • Options

      • -a archive mode: recursive, preserves symbolic links, preserves permissions, preserves timestamps, preserves owner and groups (generally a good idea to use this)
      • -E copy extended attributes, including resource forks and ACLs (mac only)
      • -e ssh indicates use a remote shell and use ssh for transfer
      • -h human readable output
      • -i view the differences (without modifying files) between destination and source
        • f indicates a file
        • s size change
        • t timestamp change
        • o owner change
        • g group change
      • -n or --dry-run to run through changes without modifying any files
      • -r recursive through folders
      • -u don't overwrite modified files in the destination
      • -v verbose, output each file
      • -W transfer the whole file that's been changed, instead of just the block(s) that have been changed
      • -z use compression (most useful for local to/from remote syncs)
      • Backups
        • --backup
        • --backup-dir
        • --suffix
      • --delete deletes files in the destination that are not in the source directory
      • --existing copy/update only the existing files in source to the destination
      • --include "[pattern]" or --exclude "[pattern]" to include/exclude certain files
        • Can be used multiple times to specify multiple files
        • --exclude-list will specify a text file with a list of exclude patterns, one on each line
      • --max-size="[size]" to set the max size of files to be transfered
      • --progress or -P to see the overall progress of the operation

• Processes

  • ps show currently running processes
    • -a show processes from all users (with open terminals)
    • -f display extra information, like uid, pid, CPU usage, etc
    • -l long listing
    • -m sort by memory usage, rather than name/id
    • -r sort by current CPU usage
    • -u display processes belonging to the specified user names
    • -x display all processes, even if they don't have a controlling terminal
    • Use with | grep [process-name] to show only the processes you are looking for
  • top displays a dynamic updated list of running processes (default sorted by pid)
    • -o [key] changes the order of the process list; keys include
      • cpu cpu usage
      • uid user id
      • user user name
  • kill [process-id] will stop a process by its id
    • −9 will force kill it by sending KILL
    • -s to send a specific signal other than TERM
  • killall [process-name] kills all instances of a process by name

• Packages

  • lsbom lists the files installed by a package
    • -f lists files
    • -l lists symbolic links
    • -s prints only the path of each file
    • -h prints the full usage of each file
  • tar [options] [source] [destination]
    • Types
      • tar is just a collection of files within a single tar file
      • tar.gz or .tgz is a compressed tar
        • Use -z option
        • Use tar then compress with gzip
    • tar -cjvf [file] [destination] creates a compressed tarfile
    • tar -tvf [file] lists contents without extracting the file
    • tar -xvf [file] extracts a file in the current directory
    • Options
      • -c create
      • -r replace
      • -x extract/restore
      • -w ask confirmation for each file
      • -v verbose
      • -j compress using bzip
      • -z compress using gzip
  • mpkg
    • xar -xvf [package] extracts a package file
    • xar -c [source] -vf [destination] compresses a package
    • Options
      • -c create/compress
      • -x extract
      • -v verbose
      • -f files
  • unzip [options] [zip-file-to-unzip] [list-of-specific-files-to-unzip] [-x files to exclude] [-d where to unzip files]
    • -f freshens existing files, extracting only files that already exist on disk and are newer
  • open [filename.zip] will unzip zip files as well

• Directories

  • cd
  • cd will take you to the previous working directory
  • cd ~
  • pushd
  • popd

• Permissions and ACLs

  • Permissions
    • d rwx rwx rwx
      • d directory, - file, l link, s socket file
      • r read (4)
      • w write (2)
      • x execute (1)
      • user (u)
      • group (g)
      • others/world (o)
    • Number of hardlinks
    • Owner and group
  • chmod [options] [permission] [filename] (to set permissions) or chmod [filename] (to see permissions)
    • Example: g+w adds write permissions to group
  • chown [user]:[group] [files]
    • Can enter user, group, or both, depending on which side of the : you specify
    • Options
      • -R recursive for all files and folders within a directory
      • -H will effect symbolic links if -R is set (not effected by default)
      • -h change the mode of the symbolic links, but not follow them
      • -L follow symbolic links (change all files and folders within links)
      • -P symbolic links are uneffected (this is the default)
      • -f do not display an error message
      • -v verbose
  • Change permissions of files or folders
    • Files only find . -type f -print0 | xargs -0 chmod 644
    • Folders only find . -type d -print0 | xargs -0 chmod 755

• Clipboard

  • pbcopy - copies contents of file into your clipboard
    • Ex: pbcopy < path/file.txt
  • pbpaste - allows use of the clipboard in terminal
    • Ex: pbpaste > path/file.txt - writes the contents of the clipboard into a file

• Searching files

  • grep -[flags] [search-pattern] [target-directory/file] (global regular expression print)
    • Used to search files with matching patterns
    • -R - search recursively
  • egrep
  • find [path] -[option] [query]
    • Time flags
      • By days
        • -mtime searches by modification time
          • find . -mtime 1 searches for all files modified exactly 1 day ago
          • find . -mtime -1 searches for all files modified within the last day
          • find . -mtime +1 searches for all files modified more than a day ago
        • -atime searches by last accessed time
        • -ctime searches by last changed time
      • By minutes
        • -cmin searches by last changed time in minutes
    • -perm searches by permission
      • find . -perm 644 finds all files with 644 permissions
    • -name
      • -iname is case insensitive
    • -delete will delete the files found using the find command
    • -type
      • find [path to install] -type d -exec chmod 755 {} \; will find all directories and change their permissions to 755
      • find [path to install] -type f -exec chmod 644 {} \; will find all files and change their permissions to 644
    • xargs
      • Used to execute other file manipulation commands on the results of a find query
      • Place after a find command and separate with |
      • find . -name "*.tmp" | xargs rm -f to remove .tmp files
      • find . -name "*.txt" | xargs grep "text" will find all text files that contain a certain text

• Search and replace

  • sed 's/[regex-to-replace]/replacement/ oldfile > newfile' - search and replace
    • Default behavior is to change only the first matching occurrence of the regex on each line
    • / is the default delimiter; any character can be used, and it's best to use a character that's not in the search or replace string (: and | are popular)
    • & can be used in the replacement string; it represents what is found in the search string
    • Add g after the replacement delimiter to replace every occurrence of the matched expression (only replaces the first occurance on each line, by default)
    • -i 'filename' to output directlyi to a file - use -i '' on osx if you want to overwrite the original file
  • perl
    • From the command line: perl -i -pe 's/search/replace/g' [filename]
      • -i read from file; use -i.bak or similar to create a backup
      • -e define perl code to be executed (in '')
      • -p adds loops around -e code
      • Search and replace options (after the s/search/replace/)
        • g - replace all occurrences of the pattern (not just the first)
        • m - search across line breaks
        • s - include line breaks as any other letter
        • i - case insensitive search
      • Search across multiple lines: perl -i -pe 'undef $/; s/search/replace/smg' [filename]

• File comparison

  • diff [file1] [file2] to compare files line by line
    • -u unified output (use this unless you have a reason not to) 
    • -i ignore case
    • -E ignore changes due to tab extension
    • -b ignore whitespace changes
    • -w ignore all white space
    • -B ignore all lines that are blank
  • Creating patches for files
    • Use diff to create a patch: diff -u [original-file] [new-file] > file.patch
    • Use patch to revert the new file to the original file: patch -R < file.patch
      • -R will reverse the patch
      • -b makes backups

• Nano text editor - nano [filename]

  • -w start nano with line wrapping off (important for configuration files)
  • Enable mouse services - Alt+M
  • Search
    • Ctrl + w search within a file
    • Alt + w find next
  • Cut/past
    • Ctrl + k cut line
    • Ctrl + u paste line
    • To cut and paste several lines, cut them successively with Ctrl + k then paste all with Ctrl + u

• Networking

  • hostname - find host/domain name and IP address
    • Displays the local hostname if not given any options
    • -d displays the domain name the machine belongs to
    • -f displays the fully qualified host and domain name
    • -i displays the IP address for the current machine
  • ping [hostname] - test network connection by sending packets of information to the defined source (also used for testing speed of connection)
    • Prefix with do to begin continuous test; use Ctrl + C to stop the test
  • ifconfig - get current network adapter configuration
  • netstat - network connections, including routing tables, interface statistics
    • -g find all the multicast groups (network) subscribed to this host
    • netstat -nap | grep [port-number] - display process id of application which is using the specified port
    • -a or -all will display all connections including TCP and UDP
    • -r for routing tables
    • --tcp or -t will display only TCP connections
    • --udp or -us will display only UCP connections
  • nslookup [hostname or ip address] - query DNS lookup name
    • Gets hostname or ip address from given ip address or hostname
  • telnet [hostname] [port] - check status of destination host
  • traceroute - view number of hops and response time to get to a remote system or web site
  • finger - view user information (user name, real name, terminal name, and write status); old command, rarely used
  • route -v add -net 10.208 -interface ppp0
    • Changes routing tables to route any ip addresses beginning with 10.208 through the VPN connection (PPP0)
  • curl - get remote files
    • -O [url] downloads a file and writes it with the same name
    • -o [filename] [url] downloads a file and lets you specify the name
  • lsof -i scan active ports for local machine
  • iptables *

• Checksums

  • md5 [filename]
  • shasum [filename] (for SHA1)
    • -a to change to 224, 256, 384, or 512

• Efficiency tips

  • Use ![command] to quickly redo the previous command (including flags and arguments)
  • Use !! for executing the previous command exactly as typed
  • Ctrl + R + [search] will search for the last command that contains whatever words you type
  • history | grep "[keyword]" searches the UNIX command history
  • Use | (pipe symbol) to separate multiple commands on the same line
  • Set up aliases in .bashrc or .profile
  • Suspending processes
    • Ctrl + Z to suspend a process
    • fg [number] or bg [number] to bring suspended processes to the foreground or background

• Users

  • adduser [username] creates new users and prompts you for the details
  • useradd creates new users and lets you manually specify each option
    • -d /home/username specifies the location of the home directory
    • -m forces the creation of the home directory
    • -g/--gid [group] lets you specify which group to add the user to (check adduser.conf to see what the default is)
    • -G/--groups [group1,group2,etc] for supplementary groups
    • --disabled-password prevents password logins (as for when only using SSH keys)
  • passwd [user] sets the password for a user -l changes the user's password to an untypable string so it can't be used to login
  • /etc/adduser.conf
    • Stores user configuration settings
    • Ubuntu defaults (change if desired)
      • All users added to their own group
      • Home user directory permissions set to 755
  • su [username] will switch to a particular user (useful to become the root user)
  • Get a list of users - cat /etc/passwd
  • Groups
    • usermod [options] [username] modifies user information
      • -g [group] change the user's initial login group
      • -a -G [group1,group2,etc] [user] will add a user to supplementary groups (the -a adds the user to those additional groups; without it, the user will be removed from any groups not specified
      • -c "Desired Name" [user] to change a users full name
    • deluser [user] [group] will delete that user from the group
    • groups [user] will display a list of groups for that user
    • groupadd and groupdel to add and remove groups
  • Sudo
    • Use sudo visudo to configure which users can use sudo
  • User shells
    • The file /etc/passwd lists each user and determines which shell they run when logged in