Skip to content

Instantly share code, notes, and snippets.

@RupGautam
Created August 25, 2017 03:59
Show Gist options
  • Select an option

  • Save RupGautam/725d5c75646238b685a465c198c64551 to your computer and use it in GitHub Desktop.

Select an option

Save RupGautam/725d5c75646238b685a465c198c64551 to your computer and use it in GitHub Desktop.

Uwinnipeg Unix Bootcamp

Lecture One

Introduction to UNIX

Operating Systems

  • allocate resources and schedule tasks
  • resources include CPU, memory, disk, tape, printers, terminals, modems, etc.
  • only one user allowed at a time for each resource
  • keeps track of filenames and directory structure
  • multi-tasking (one task at a time per processor actually executing)
  • multi-processing (schedules multiple processors)
  • multi-user

History of UNIX

  • developed at Bell Labs (AT&T) in 1969
  • unlike most OS's at the time, UNIX was multi-user, interactive, and simplified sharing of data & programs
  • became popular in industry as college and university graduates were trained in it
  • open system, ported to many different hardware platforms (unlike IBM, Burroughs, Univac mainframes)
  • software written for one UNIX system will often run on other systems with little or no modification
  • still very weak in areas of system management - tape management, security, hardware accounting, capacity planning tools, performance management including prioritization of jobs
  • hardware manufacturers modified UNIX to run on their systems and added enhancements
  • standardization was begun in response to Windows NT threat
  • System V Release 4 is one of the steps to address standardization
  • SVR4 includes many of the modifications that were being done by hardware manufacturers, if they were universally useful and applicable

History of Linux

  • GNU (Gnu's not Unix) was started in early 80's to create and promote free software
  • Linux was created by Linus Torvalds in the 1990's, released to the Internet in 1994, mostly using GNU C compiler
  • about 1/3 of Linux is GNU code from the Free Software Foundation - a Linux distribution consists of Linux kernel + GNU compilers/tools/utilities + other free software
  • Linux uses the Open Source model for development - code is placed on the Internet, users download and test it, programmers improve it and place it back on the web
  • there is competition among programmers to fix bugs and improve Linux

UNIX Structure

  • Hardware, surrounded by
  • UNIX Kernel (basic OS), surrounded by
  • Shell (user interface, command interpreter, and some built-in commands), surrounded by
  • Utilities (or commands)
  • most common shells: Bourne shell (sh), C shell (csh), Korn shell (ksh), Bourne again shell (bash), TC shell (tcsh), Z shell (zsh)
  • we'll mostly be concerned with the Bash shell, which is the most popular Linux shell
  • the Korn shell is the most popular Unix shell, and is very similar to the Bash shell

Entering Commands, Logging In & Out

  • to login, use an ssh (or similar) program, and provide userid and password
  • to backspace (erase), usually backspace, control-h, or control-backspace
  • to interrupt a process, usually control-c
  • to logout, use exit, logout, or control-d
  • use passwd to change password
  • command line arguments can be separated by one or more spaces or tabs

Some Commands

  • pwd will show your current directory
  • cd is used to change current directory, eg. cd directory-name
  • ls - lists information about files and directories
  • ls -a - all files (including hidden)
  • ls -l - long form, gives more information about files
  • ls -d - gives information about the directory itself, not contained files
  • these options can be mixed and matched, eg. ls -ld
  • cal - displays a calendar of the current month
  • cal 1995 - displays calendar for specified year
  • cal 3 1995 - displays calendar for specified month of specified year
  • date - gives date and time
  • who, w - information about users logged on to system
  • who am i, whoami - information about your session

####File System; Common Commands;

File System

  • Unix file system is hierarchical (tree, directory within directory)
  • root directory is / - ancestor of all files and directories on system
  • directories "contain" files and/or other directories
  • directories actually only contain filenames and pointers to inodes (information about the files)
  • to be safe, for file and directory names use alphabetics, numerics, underscore, period, or comma
  • some UNIX systems allow up to 255 characters in a filename, others only allow up to 14
  • files within one directory must have different names
  • hidden filenames begin with a period, ls won't list them, but ls -a will
  • hidden filenames are used mostly for configuration files, that you don't want to see every time you issue an ls command
  • . represents the current working directory, .. represents its parent
  • home directory is the working directory upon log in, or after the cd command is issued with no arguments

Standard Directories

  • standard directories & files
  • /home typically contains the home directories of all users
  • /bin and /usr/bin contain standard utility programs
  • /sbin and /usr/sbin contain utilities for system administration
  • /etc contains admin & configuration files, such as /etc/passwd
  • /var contains files that vary as the system is running, such as temp, log, spool, and mailbox files
  • /dev contains files representing peripheral devices (device drivers)
  • /tmp is used by some programs for temporary files

File Types

  • file types, as indicated by the first character in ls -l output:
  • - indicates an ordinary file
  • d indicates a directory
  • l indicates a symbolic link
  • b indicates a block device file, such as for a dvd reader
  • c indicates a character device file, such as for a keyboard
  • p indicates a pipe, used to communicate between processes on the same server
  • s indicates a socket, used to communicate between processes on the same or different servers

Common Commands

  • touch - creates a new file, or updates stats on an existing file
  • mkdir - to create directories
  • mkdir -p - to create a directory & any parent directories not already existing
  • rmdir - to delete empty directories, eg. rmdir directory-list
  • mv existing-file new-file - to rename or move files & directories
  • cp source-file destination-file - to copy files (careful, overwrites destination with no warning)
  • cp -r - to copy directories including files and subdirectories
  • rm - to delete files, eg. rm file-list
  • rm -r - to delete directories including files and subdirectories
  • rm -ir - to delete directories including files and subdirectories, with prompt to confirm removal
  • cat filename - display contents of file
  • more or less - displays file contents one page at a time
  • - will go to next page
  • b - will go to previous page
  • /string - will search for string within document being viewed
  • q - will quit
  • file filename - gives info about the contents of the file
  • find - to find files matching specified characteristics
  • find . -name file* - lists pathname of any filenames beginning with "file", from the current directory and any subdirectories
  • find . -size +50k - lists pathname of any files larger than 50 kb, from the current directory and any subdirectories
  • man command - online manual (or help) for command, uses more to display information
  • man -k keyword - eg. man -k calendar - searches through man sections for keyword
  • diff file1 file2 - displays differences between 2 files
  • echo text or $variable (eg. $HOME)
  • which utility - lists pathname that would be used to access this utility

###Pathnames; Filename Expansion; Shell Basics

Pathnames

  • absolute pathnames start with /, representing the root, and continue with the directory structure to the destination filename or directory name
  • relative pathnames do not start with /, and are relative to your current (or working) directory
  • relative-to-home pathnames are actually absolute, but use ~ to represent the home directory
  • replaced by $HOME, eg. /pathname
  • ~user replaced by home directory of user, eg. ~user/pathname

Filename Expansion

  • also called ambiguous file references, metacharacters, wild card characters, and filename generation characters
  • used to find filenames that match a pattern
  • ? matches any single character, eg. echo temp?2
  • matches any number (including none) of characters, eg. ls temp
  • a leading period (hidden file) must be explicitly specified, eg. echo * - will not show hidden files
  • [ ] matches any single character in included list, eg. ls temp[12]*
  • - within [ ] between two characters represents a range, eg. ls temp[1-58] is the same as ls temp[123458]
  • if ! is first character within [ ], then any character not in list is matched

Shell Basics

  • command history
  • or - move to previous command or next command
  • fc -l - display last 16 commands
  • history - display all commands in buffer
  • !num - re-execute command number "num"
  • !xxx - re-execute last command beginning with string "xxx"
  • some useful BASH keyboard shortcuts:
  • erase Characters: Backspace or Ctrl-Backspace or Ctrl-h
  • go to the beginning of the line: Ctrl-a
  • go to the end of the line: Ctrl-e
  • delete a word before the cursor: Ctrl-w
  • delete everything before the cursor: Ctrl-u
  • clear screen: Ctrl-l
  • search for a keyword in previous commands: Ctrl+r
  • auto-complete file/directory names: Tab

Lecture Five

Simple Filter Commands; Redirection; Piping; Multiple Commands

Simple Filter Commands

  • head -7 filename - displays first 7 lines, 10 is default
  • tail -11 filename - displays last 11 lines, 10 is default
  • cut is used to extract fields and characters from records
  • cut -f2 filename - extract 2nd field from all records in file1, using tab as delimiter (default)
  • cut -d' ' -f2,5 filename - extract 2nd and 5th field, using space as delimiter
  • cut -d' ' -f1-3,5 filename - extract 1st through 3rd and 5th fields, using space as delimiter
  • cut -c3-5 filename - extract 3rd to 5th characters
  • sort filename - displays records in ascending order
  • by default uses dictionary (ascii sort) order, from first to last character in each record
  • sort -f filename - sort ignoring case (fold to uppercase)
  • sort -k 3 filename - sort on 3rd field (default field delimiter is space)
  • sort -t: -k 3 filename - sort on 3rd field using colon as field delimiter
  • sort -rk 3 filename - sort on 3rd field in reverse (descending) order
  • sort -nk 5 filename - sort numerically on 5th field
  • sort -u filename - sort records, drop duplicate records
  • tr is used to translate characters to different characters
  • tr a A < filename - translate all characters "a" to "A"
  • tr "[a-z]" "[A-Z]" < filename - translate lowercase "a" through "z" to uppercase
  • tr ':' ' ' < filename - translate all colons to spaces
  • tr ' ' '\n' < filename - translate all spaces to newline characters
  • tr -d '\n' < filename - delete all newline characters
  • wc filename - displays various counts of the contents of a file
  • wc -l filename - displays number of lines in file
  • wc -c filename - displays number of characters in file
  • wc -w filename - displays number of words in file
  • grep 'string' filename - displays lines in file that contain the string
  • grep will be revisited in the lecture about regular expressions

Redirection

Standard Input, Output, and Error

  • standard input, output, and error for commands are sent to your terminal, unless they are redirected
  • cat - (by itself) will take input from terminal (use control-d to end input), send output and error messages to terminal
  • will redirect standard output to a file, deleting any existing contents of the file

  • cat > temp - will redirect output to a file called temp (this technique is sometimes called an "on the fly" document)
  • will redirect standard output to a file, but will append (add to) the end of the file

  • cat temp >> temp2 will append the contents of temp to the contents of temp2
  • 2> will redirect standard error to a file (note that > is the same as 1>)
  • 2>> will append standard error to a file
  • &2 (or 1>&2) will redirect standard output to standard error

  • /dev/null can be used to get rid of unwanted output
  • find / -name *.tmp 2>/dev/null
  • < will redirect standard input from a file, as in previous examples of tr

Piping

  • | (pipe) will connect the standard output of the command to its left, to the standard input of the command to its right
  • ls -al | more
  • ls -al | grep "temp" | sort -rnk5
  • tee will take standard input from a pipe, and send it as output to one or more files and to its standard output
  • ls -al | tee file1 file2

Multiple Commands

  • besides piping, there are other ways that multiple commands may be placed in one line
  • commands may be separated by semi-colons
  • each command will be executed when the previous command has terminated
  • for example: sleep 5; ls
  • commands may be grouped by using parentheses
  • (echo "Who is on:"; w) > whoson
  • commands may also be split over multiple lines, making it easier (for humans) to interpret a long command
  • quote or "escape" the newline character at the end of a line, to get rid of the special meaning of newline (to end a command line)
  • for example:

echo "This will be split over multiple
lines. Note that the shell will realize
that a pipe requires another command, so
it will automatically go to the next line" | tr '[a-z]' '[A-Z]'

####Linking; Process Management; Shell Startup; Command History; alias

Linking

  • ln (link) command gives a file an additional name, or pointer, from the same or a different directory
  • one common use of links is to give a long pathname an easy to remember shortcut
  • hard links
  • ln file1 file2 - gives file1 the additional name file2 (hard link)
  • ls -i file1 - and
  • ls -i file2 - will give same index (or inode) numbers, they are physically the same file
  • a file will not be removed until all of its hard links are removed (using rm)
  • symbolic links (or soft links)
  • ln -s file1 file2 - gives file1 the additional name file2, as an indirect pointer
  • a file can be removed even if it still has symbolic links
  • ln -s path/dir1 dir2 - gives dir1 the additional name dir2 in the current directory
  • directories can only be linked symbolically (except by the administrator)
  • symbolic links are useful where you wish to recreate a file without having to remove and recreate a lot of links

Process Management

  • a process is the execution of a command
  • every process has a process id
  • a parent process forks (or spawns) a child process
  • & at the end of a command line will run the command in the background, and gives the unique process identification (PID) and a job number
  • eg. ls -lR / > test 2> testerror & (recursive listing of all files and directories under the root directory)
  • eg. tail -f test & (endless loop waiting for additional lines)
  • ps - will display process status with PID, for current terminal only
  • ps -U username - will display status for all terminals belonging to specified username
  • ps -f - will display additional info, such as parent PID (PPID) of each process
  • ps f - will display processes showing parent/child relationships
  • there are LOTS of ps options, varies by system
  • top - will display resource usage, continually updated (end with 'q')
  • jobs - will show job number of processes
  • kill pid# (or %job) - will abort process with specific PID or job number
  • kill -9 pid# - will abort process that has been written to ignore other signals
  • kill -stop pid# - will suspend (stop) a background job
  • -z - will suspend a foreground job
  • bg %job - will start running a suspended process in the background, defaults to last suspended job
  • fg %job - will start running a suspended process in the foreground

Shell Startup

bash startup

  • upon login, /etc/profile executes
  • controlled by system administrator
  • common to all shells
  • then ~/.bash_profile executes
  • used to customize environment for each user (eg. set PATH, umask, mesg, aliases, etc.)
  • .bash_login is used if .bash_profile doesn't exist
  • .profile is used if .bash_profile and .bash_login don't exist
  • .bashrc is used for interactive sub-shells
  • .bash_logout is executed when shell is exited

ksh startup

  • upon login, /etc/profile executes, then .profile in home directory
  • .kshrc is used for interactive sub-shells

Command History

  • .bash_history is used to store recently executed command lines
  • allows use of arrow keys to move back and forth within history
  • -r can be used to search history by keyword

alias

  • assigns a new name to an existing utility
  • eg. alias dir=ls
  • eg. alias ls='ls -al'
  • eg. alias clearfile='cat /dev/null >'
  • can be useful, but can also make your scripts cryptic
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment