Skip to content

Instantly share code, notes, and snippets.

@imShakil

imShakil/lpi101.md

Forked from mbodo/lpi101.md
Created August 2, 2023 15:57
Show Gist options
  • Save imShakil/d336e92f212f9d6ddfe8dc1f82cb010e to your computer and use it in GitHub Desktop.
Save imShakil/d336e92f212f9d6ddfe8dc1f82cb010e to your computer and use it in GitHub Desktop.
Download ZIP
LPIC-1 Exam 101
Raw
lpi101.md

LPI-1 101 Notices

Topic 101: System Architecture

Topic 102: Linux installation and package management

Topic 103: GNU and UNIX commands

Topic 104: Devices, Linux Filesystems, Filesystem Hierarchy Standard

101.1 Determine and configure hardware settings

  • IRQ - Interrupt request
  cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:        118          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  1:          0          0          0         11          0          0          0          0  IR-IO-APIC   1-edge      i8042
  8:          0          1          0          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0        253          0          0          0          0          0          0  IR-IO-APIC   9-fasteoi   acpi
 12:          0          0        149          0          0          0          0          0  IR-IO-APIC  12-edge      i8042
 16:          0          0          0          0          0         29          0          0  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
 17:          0          0          0          0          0          0        959          0  IR-IO-APIC  17-fasteoi   snd_hda_intel:card1
 18:          0          0          0          0          0          0          0          0  IR-IO-APIC  18-fasteoi   i801_smbus
 23:          0          0          0          0          0          0         35          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0    9705150          0          0          0  IR-PCI-MSI 327680-edge      xhci_hcd
 26:          0          0          0          0          0          0          0     262407  IR-PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 27:          0          0          0         11          0          0          0          0  IR-PCI-MSI 360448-edge      mei_me
 28:          0          0          0          0    1826232          0          0          0  IR-PCI-MSI 31457280-edge      enp60s0
 29:          0          0          0          0          0       1074          0          0  IR-PCI-MSI 442368-edge      snd_hda_intel:card0
 30:          0          0          0          0          0          0          0       3990  IR-PCI-MSI 30932992-edge      iwlwifi
 31:    2178755          0          0          0          0          0          0          0  IR-PCI-MSI 524288-edge      nvidia
...

Legend/Example:

0-31 - IRQ
CPU0-7 - number of interrupts per IRQ
IR-PCI-MSI 31457280-edge - interrupt type
enp60s0 - device / kernel module
  • I/O Addresses
cat /proc/ioports

0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
  • DMA Addresses
cat /proc/dma

4: cascade

  • /sys - sysfs virtual file system

    • files represent object attributes
    • directories represent object themselves
    • top-level directories represent major subsystems
    • symbolic links represent relationships between objects
    mount sysfs

mount -t sysfs sysfs /sys

  • HAL - Hardware abstraction level (replaced by udev)

    Links:

  • D-Bus - Desktop Bus

    Links:

  • udev - responsible for the dynamic device management needed for hot plugging devices

    • information about configured and active devices is contained in the /dev
      • b - block device
      • c - character device
      • d - directory
      • l - symbolic link to (dev,proc,run)
    • create device files 
      mknod /dev/sda1 b 8 2

b - block device
8 - major number
2 - minor number
    • daemon systemd-udevd.service, manage udev events
    • default udev rules /usr/lib/udev/rules.d
    • local udev rules (may override defaults) /etc/udev/rules.d
    • udev configure /etc/udev/udev.config
    • udevadm program is an administration tool for udevd e.g. 
      udevadm info --query=all –-name=/dev/sda

P: /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NSAG521465L
E: DEVLINKS=/dev/disk/by-id/wwn-0x50025388a0a87a4c /dev/disk/by-path/pci-0000:00:1f.2-ata-1 /dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S250NSAG521465L

P - sysfs device path
N - device node (in the /dev)
S - symbolic link, placed by udevd in /dev
E - additional device info from udevd rules
    • monitor udevadm events 
      Monitor messages, incoming from kernel, udevd sends out to other programs

udevadm monitor 

Only kernel and match scsi only

udevadm monitor --kernel --subsystem-match=scsi

    Links:

  • lpci - list all PCI devices

    • looks up to /usr/share/hwdata/pci.ids
    • new device, query PCI ID server database using DNS lookup
    Query and write to ~/pciids-cache

lspci -q 

    Detailed pci list

lspci -vvv

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

    Show names and numbers

lspci -nn

...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204M [GeForce GTX 965M] [10de:13d9] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
...

    Tree of devices in slot 00

lspci -tvv -s \:00

-+-[0000:3c]---00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
 +-[0000:3b]---00.0  Intel Corporation Wireless 7265
 +-[0000:01]-+-00.0  NVIDIA Corporation GM204M [GeForce GTX 965M]
 |           \-00.1  NVIDIA Corporation GM204 High Definition Audio Controller
 \-[0000:00]---00.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller

    List by vedor id (in this example Nvidia 10de)

lspci -d 10de\:

01:00.0 VGA compatible controller: NVIDIA Corporation GM204M [GeForce GTX 965M] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

  • lsusb - list all USB buses/devices

    • support USB 1,2,3
    • USB 1.1 (12Mbps), driver OHCI(UHCI)
    • USB 2.0 (480Mbps), driver EHCI
    • USB 3.0 (5Gbps), driver XHCI
    lsusb

Bus 002 Device 002: ID 8087:8000 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp.
...

    Tree view

lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/14p, 480M
  |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
  |__ Port 2: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
...

    Device ID view

lsusb -d 0951\:

Bus 003 Device 004: ID 0951:1660 Kingston Technology Data Traveller 108

  • Kernel modules

    • lsmod - basically it formats /proc/modules
    Module                  Size  Used by
usbcore               286720  9 uvcvideo,usbhid,snd_usb_audio,ehci_hcd,xhci_pci,snd_usbmidi_lib,btusb,xhci_hcd,ehci_pci
usb_common             16384  1 usbcore
i8042                  32768  1 asus_nb_wmi
serio                  28672  9 serio_raw,atkbd,psmouse,i804

Used by: 
usb_common 16384  1 usbcore

[usb_common] - actual module
[1] - number of process, other modules which are using the kernel module (in this case usb_common module)
[usbcore] - relies on usb_common
    • modinfo - show info about kernel module
    modinfo serio

filename:       /lib/modules/4.15.6-1-ARCH/kernel/drivers/input/serio/serio.ko.xz
license:        GPL
description:    Serio abstraction core
author:         Vojtech Pavlik <[email protected]>
srcversion:     91E703BAD5EBF435706CF36
depends:
retpoline:      Y
intree:         Y
name:           serio
vermagic:       4.15.6-1-ARCH SMP preempt mod_unload modversions

    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204M [GeForce GTX 965M] [10de:13d9] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

Usefull for scripts:

modinfo -F vermagic serio
4.15.6-1-ARCH SMP preempt mod_unload modversions

modinfo -F filename serio
/lib/modules/4.15.6-1-ARCH/kernel/drivers/input/serio/serio.ko.xz
    • insmod - loads module only, not dependencies (it expects they are already loaded)
    insmod /lib/modules/4.15.6-1-ARCH/kernel/net/ipv4/netfilter/ip_tables.ko.xz
    • modprobe - Add or remove kernel modules, automatically loads any depended-on module
    Load the ip_tables module and dependecies:

modprobe ip_tables

    Dry run with verbose mode, perform checks except module insertions:

modprobe -v --dry-run ip_tables

    modprobe --show-depends nfsd
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/net/sunrpc/sunrpc.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfs_common/grace.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/lockd/lockd.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfs_common/nfs_acl.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko.xz 
insmod /lib/modules/3.10.0-693.11.6.el7.x86_64/kernel/fs/nfsd/nfsd.ko.xz 

    Remove module:

modprobe -r ip_tables
    • rmmod - remove kernel module, operates on single module, if depend on other module or in use it will return error message
    Remove kernel module:

rmmod ip_tables

Remove module with wait util become unused:

rmmod -w ip_tables

    Links

Links Configure hardware settings

101.2 Boot the system

  • Boot sequence

    • MBR single OS
      • BIOS (Basic Input Output Service), stored on non-volatile ROM,EEPROM
      • BIOS after turn on, power-on self test (POST)
      • Loads first sequence from MBR (Master Boot Record)
      • MBR contains the partition table
      • The code loaded by BIOS from first sector is first stage boot loader
      • Check for primary partition marked as active, partition boot record
      • Load the OS, this is stage 2 boot loader
      • Notice: MBR system cannot boot from logical partition, only primary and active

  • Common linux bootloaders

    • LILO, the LInux LOader
    • GRUB, GRand Unified Boot loader (GRUB legacy)
    • GRUB2
    • Syslinux

  • Chain loading

    • boot manager could load another boot manager
    • e.g LILO in one partition chain load GRUB in another partition

  • EFI and UEFI

    • UEFI, Unified Extensible Firmware Interface
    • 32bit, 64bit
    • Many, not all UEFI impl. allow booting in UEFI mode or Legacy

  • GRUB2 Menu

    • e - edit
    • c - grub command line

  • Kernel parameters

    • single user mode

  • The init process

    • when kernel finish loading, it usually starts /sbin/init
    • int always assigned to PID 1
    • System V / Upstart, /sbin/init
    • systemd, /sbin/init -> /lib/systemd/systemd

  • System V

    • runlevels
      • 0 : poweroff (halt)
      • 1 : rescue (single user mode), aliased as s or S
      • 2 : multi-user mode without networking
      • 3 : multi-user with networking
      • 5 : multi-user with networking and the X Window System
      • 6 : reboot
    • /etc/rc.d/init.d, /etc/init.d
    • after completion, init start getty in case of headless/console mode, or graphical login screen

  • Systemd targets (runlevels)

    /usr/lib/systemd/system/runlevel0.target -> poweroff.target
/usr/lib/systemd/system/runlevel1.target -> rescue.target
/usr/lib/systemd/system/runlevel2.target -> multi-user.target
/usr/lib/systemd/system/runlevel3.target -> multi-user.target
/usr/lib/systemd/system/runlevel4.target -> multi-user.target
/usr/lib/systemd/system/runlevel5.target -> graphical.target
/usr/lib/systemd/system/runlevel6.target -> reboot.target

  • Init RAM disk

    • to list ram disk (CentOS 7), lsinitrd /boot/initramfs-3.10.0-693.11.6.el7.x86_64.img

  • Boot events

    • dmesg, kernel ring buffer
    dmesg | head -n 30
    • kernel ring buffer is used also after system booted, e.g hot plug events, program failures

  • Logging systems messages

    • System V init, /var/log/messages
    • systemd, journalctl -xb (display entries from the current boot)

    Links

101.3 Change runlevels / boot targets and shutdown or reboot system

  • System V

    # Default runlevel. (Do not set to 0 or 6)
id:3:initdefault:
    • Changing runlevels
      • pernament, through /etc/inittab
      • boot, grub e.g pass 3 to kernel params
      • immediatelly telinit, /sbin/telinit -> init
    • Explore runlevel
      • runlevel 
        runlevel
N 3

N - previous runlevel, N means runlevel has not been changed since the system was booted
3 - current runlevel

  • Single-user mode

    • telinit 1, telinit s

  • Clear shutdown

    • shutdown [OPTIONS...] [TIME] [WALL...]
    • shutdown command, it shutdown or reboot system, the init process sends all running processes SIGTERM signal and after 5 seconds delay init sends a SIGKILL signal
    • by default shutdown switches to runlevel 1, or to runlevel specified by -h halt or -r reboot
    • [TIME] e.g 23:59, relative 10m, immediate +0 alias for now
    • [WALL] message to be displayed 15 min before shutdown command is executed, if [TIME] is < 15 min then message is displayed immediately
    • -t60 delay SIGKILL signal after SIGTERM signal sent to processes, see above
    • shutdown -c cancel the shutdown command
    • if [TIME] is used, 5 minutes before the system goes down the /run/nologin file is created to ensure that further logins shall not be allowed.

  • Notify users with wall

    $ wall Scheduled outage at 23:59 has been canceled

    Scheduled outage at 23:59 has been canceled
 
$ echo -e "We are experiencing system problemsOutage rescheduled to 02:30" | wall

     We are experiencing system problemsOutage rescheduled to 02:30

  • Halt, reboot, and poweroff

    • halt
    • poweroff, is symbolic link to halt 
      /sbin/poweroff -> halt
    • reboot, is symbolic link to halt 
      /sbin/reboot -> halt

  • System V /etc/inittab

    • /etc/inittab
    • id:runlevels:action:process
    • action:
      • respawn (getty)
      • wait
      • once
      • initdedault
      • ctrlaltdel
    • e.g 
      c1:12345:respawn:/sbin/agetty 38400 tty1 linux

ca::ctrlaltdel:/sbin/shutdown -t5 -r now

  • System V Initialization scripts

    • runlevels in /etc 
      rc -> rc.d/rc
rc0.d -> rc.d/rc0.d
rc1.d -> rc.d/rc1.d
rc2.d -> rc.d/rc2.d
rc3.d -> rc.d/rc3.d
rc4.d -> rc.d/rc4.d
rc5.d -> rc.d/rc5.d
rc6.d -> rc.d/rc6.d
rc.d/rc.local
rc.d/rc.sysinit
    • (centos 5) scripts are implemented in /etc/init.d/
    • script for runlevel is than a symbolic link to /etc/rc[0-6].d/ with pattern:
      • Snn (e.g. S27auditd) - for starting
      • Knn (e.g. K72auditd) - for switch off the service
    cd /etc/rc5.d

S27auditd -> ../init.d/auditd

cd /etc/rc0.d

S72auditd -> ../init.d/auditd

  • Upstart

    • driven by events
    • Events are used to trigger tasks or services, collectively known as jobs

  • Systemd

    • units, which can be grouped into control groups or cgroups
      • service units
      • socket units
      • device units
      • mount units
      • automount units
      • target unit
      • snapshot units

Links

102.3 Manage shared libraries

  • Static and dynamic linking

    • Statically linked executables, /sbin/sln
    • Dynamically linked executables, /bin/ln

  • The ldd command

    • displays shared object dependencies of executable if any exists 
      $ ldd /sbin/sln
not a dynamic executable

$ ldd /bin/ln
linux-vdso.so.1 (0x00007fff9c8c6000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fc5c7ae0000)
  /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fc5c80a8000)

  • Dynamic library configuration

    • /etc/ld.so.conf, configuration
    • /etc/ld.so.conf.d, partial configuration read by ld.so.conf
    • /etc/ld.so.cache, used by dynamic loader
    • if /etc/ld.so.conf//etc/ld.so.conf.d were changed them ldconfig must run to rebuild /etc/ld.so.cache
    • to display ld.so.cache 
      $ /sbin/ldconfig -p

  • Loading specific libraries

    • e.g support old dynamic shared library, use 
      $ export LD_LIBRARY_PATH=/usr/lib/oldstuff:/opt/libs

Links: -software-interrupts-vs-system-calls

  • PATA Disks

    • Master - /dev/hda
    • Slave - /dev/hdb
    • Partitions - /dev/hda1, /dev/hda2

  • SCSI Disks

    • Hard drives /dev/sda,/dev/sdb,..
    • CD/DVD ROM /dev/scdx,/dev/srx
    • Tape /dev/stx,/dev/nstx

102.4 Use Debian package management

  • Installing Debian packages 
    sudo apt-get install openjdk-7-jre-headless
  • Simulated or dry-run install 
    sudo apt-get install -s openjdk-7-jre-headless

  • Package locations

    • Packages resources 
      cat /etc/apt/sources.list
    • Additional Packages resources configuration 
      /etc/apt/sources.list.d
    • To synchronize local database of packages, should run befor every package install or update 
      apt-get update

  • Removing Debian packages

    remove - remove package, but configuration files persists in system

apt-get remove -s openjdk-7-jre-headless
    • autoremove, will removes also installed dependencies when the are no longer required 
      apt-get autoremove -s openjdk-7-jre-headless

apt-get remove auto-remove -s openjdk-7-jre-headless
    • autoremove whithout package name will remove all unused packages 
      apt-get autoremove
    • purge, remove package and configuration files 
      apt-get purge

  • Updating Debian packages

    prerequisite packages will also be upgraded

sudo apt-get upgrade ghostscript

  • Updating all packages or upgrading to a new distribution

    upgrade all packages

sudo apt-get upgrade

upgrade distribution

sudo apt-get dist-upgrade

  • APT configuration

    /etc/apt/apt.conf

  • Reconfiguring Debian packages

    • debconf, re-configure packages after they are installed 
      dpkg-reconfigure tzdata
    • debconf-show, displays current configuration 
      debconf-show tzdata

debconf-show adduser

  • Package status with dpkg

    • configuration wide /etc/dpkg/dpkg.cfg
    • configuration user ~/.dpkg.cfg
    • dpkg status of packages 
      $ dpkg -s gfortran tzdata

Package `gfortran' is not installed and no info is available.

Package: tzdata
Status: install ok installed
Multi-Arch: foreign
...

  • Packages and the files in them

    • to list files(including directories) of package 
      dpkg -L gcc

/.
/usr
/usr/bin
/usr/bin/c89-gcc
/usr/bin/c99-gcc
...

  • Find Package which contains a file

    dpkg -S /usr/lib/libmagic.so.1

libmagic1: /usr/lib/libmagic.so.1

  • Alternatives

    • for packages like java, openJDK etc.
    • update-alternatives
    ls -la /etc/alternatives

e.g. to display actual command link

update-alternatives --display java

  • Aptitude

    • ncurses package manager
    • C-T access to menu
    • / search for package

  • Finding Debian packages

    apt-cache search "linux loader"

  • Download only mode

    apt-get install -d gfortran

than list in

ls -la /var/cache/apt/archives/gfo*

/var/cache/apt/archives/gfortran_4%3a4.6.3-1ubuntu5_amd64.deb

than list contents with

dpkg -c /var/cache/apt/archives/gfortran_4%3a4.6.3-1ubuntu5_amd64.deb | grep -i --color 'bin'

./usr/bin/
./usr/bin/x86_64-linux-gnu-gfortran -> gfortran-4.6
./usr/bin/gfortran -> gfortran-4.6

Links - what-is-the-difference-between-apt-get-and-dpkg

102.5 Use RPM and YUM package management

  • Installing RPM packages

    • package management system used for packaging in the Linux Standard Base (LSB)
    • three subgroups:
      • Querying and verifying packages
      • Installing, upgrading, and removing packages
      • Performing miscellaneous functions
    RPM install doesn't resolve dependencies, it warn but they must be downloaded and installed

rpm -i gcc-gfortran-4.8.5-16.el7.x86_64.rpm
warning: gcc-gfortran-4.8.5-16.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
error: Failed dependencies:
gcc = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libgfortran = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libgfortran.so.3()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libmpc.so.3()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libmpfr.so.4()(64bit) is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libquadmath = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
libquadmath-devel = 4.8.5-16.el7 is needed by gcc-gfortran-4.8.5-16.el7.x86_64
    • YUM install of RPM packages
    YUM resolve dependencies automatically

yum install gcc-gfortran

  • Package locations

    • repositories /etc/yum.repos.d/
    • YUM configuration file /etc/yum.conf
    • repo file is by default divided to 3 sections:
      • normal packages
      • debug packages
      • source packages
    • yum clean, clean locally stored packages database
    • yum makecache, create locally stored packages database

  • Removing RPM packages

    • RPM remove test run 
      RPM system does not maintain information on packages that were automatically added

sudo rpm -e --test vim-minimal
error: Failed dependencies:
    vim-minimal is needed by (installed) sudo-1.8.19p2-11.el7_4.x86_64
    • YUM will offer to remove also dependent packages 
      sudo yum remove vim-minimal

Removing:
vim-minimal
Removing for dependencies:
sudo

Remove  1 Package (+1 Dependent package)

  • Upgrading RPM packages

    • RPM upgrade 
      Download:

cairo-1.14.2-1.fc21.x86_64.rpm
cairo-gobject-1.14.2-1.fc21.rpm

cd to:

rpm -Uvh *.rpm

Preparing...                          ################################# [100%]
Updating / installing...
1:cairo-1.14.2-1.fc21              ################################# [ 25%]
2:cairo-gobject-1.14.2-1.fc21      ################################# [ 50%]
Cleaning up / removing...
3:cairo-gobject-1.13.1-0.4.git337ab################################# [ 75%]
4:cairo-1.13.1-0.4.git337ab1f.fc21 ################################# [100%]

Where:
- U upgrade/install packages
- v verbose
- h display hashmarks as progress bar
    • Yum upgrade 
      yum update 'pop*'
Loaded plugins: langpacks
Resolving Dependencies
--> Running transaction check
---> Package poppler.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler.x86_64 0:0.26.2-9.fc21 will be an update
---> Package poppler-data.noarch 0:0.4.7-1.fc21 will be updated
---> Package poppler-data.noarch 0:0.4.7-2.fc21 will be an update
---> Package poppler-glib.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler-glib.x86_64 0:0.26.2-9.fc21 will be an update
---> Package poppler-utils.x86_64 0:0.26.2-3.fc21 will be updated
---> Package poppler-utils.x86_64 0:0.26.2-9.fc21 will be an update
...

  • Querying RPM packages

    • RPM basic query 
      rpm -q vim-minimal

vim-minimal-7.4.160-2.el7.x86_64
    • RPM detailed query 
      rpm -qi vim-minimal

Name        : vim-minimal
Epoch       : 2
Version     : 7.4.160
Release     : 2.el7
    • yum info 
      yum info vim-minimal

Installed Packages
Name        : vim-minimal
Arch        : x86_64
Epoch       : 2
...
    • rpm --queryformat, to build custom query output
    • rpm --querytags, to list all tags supported of rpm version

  • RPM packages and files in them

    • RPM query files in package (package must be installed) 
      rpm -ql vim-minimal

/etc/virc
/usr/bin/ex
/usr/bin/rvi
/usr/bin/rview
...
    • RPM query configuration files in package (package must be installed) 
      rpm -qc vim-minimal

/etc/virc
    • RPM query documentation files in package (package must be installed) 
      rpm -qd vim-minimal

/usr/share/man/man1/ex.1.gz
/usr/share/man/man1/rvi.1.gz
/usr/share/man/man1/rview.1.gz
/usr/share/man/man1/vi.1.gz
/usr/share/man/man1/view.1.gz
/usr/share/man/man1/vim.1.gz
/usr/share/man/man5/virc.5.gz
    • RPM query files in package (not installed only downloaded) 
      rpm -qpl gcc-gfortran-4.8.5-16.el7.x86_64.rpm    

/usr/bin/f95
/usr/lib/gcc
/usr/lib/gfortran
/usr/lib/gcc/x86_64-redhat-linux
    • RPM query documentation files in package (not installed only downloaded) 
      rpm -qpd gcc-gfortran-4.8.5-16.el7.x86_64.rpm

/usr/share/info/gfortran.info.gz
/usr/share/man/man1/gfortran.1.gz
...

  • Querying all installed packages

    List all packages sorted

rpm -qa | sort | more

Count all packages in system

rpm -qa | wc -l

Count all files in installed system packages

rpm -qal | wc -l

  • Querying which package owns a file

    $ which vi

/usr/bin/vi

$ rpm -qf $(which vi)

vim-minimal-7.4.160-2.el7.x86_64

  • RPM dependencies

    • RPM query package dependencies (not installed, download only) 
      $ rpm -qpR gcc-gfortran-4.8.5-16.el7.x86_64.rpm

/bin/sh
/bin/sh
/sbin/install-info
/sbin/install-info
gcc = 4.8.5-16.el7
ld-linux-x86-64.so.2()(64bit)
...
    • RPM query package dependencies (package is installed) 
      $ rpm -qR vim-minimal

config(vim-minimal) = 2:7.4.160-2.el7
libacl.so.1()(64bit)
libacl.so.1(ACL_1.0)(64bit)
libc.so.6()(64bit)
...

  • Yum list package dependencies

    • package dependencies with specified package architecture
    yum deplist $(rpm -q vim-minimal)

package: vim-minimal.x86_64 2:7.4.160-2.el7
  dependency: libacl.so.1()(64bit)
    provider: libacl.x86_64 2.2.51-12.el7
  dependency: libacl.so.1(ACL_1.0)(64bit)
    provider: libacl.x86_64 2.2.51-12.el7
  dependency: libc.so.6(GLIBC_2.15)(64bit)
    provider: glibc.x86_64 2.17-196.el7_4.2
  ...
    • package dependencies without specified package architecture (display x86_64, i686) 
      $ yum deplist vim-minimal

...
  dependency: libtinfo.so.5()(64bit)
    provider: ncurses-libs.x86_64 5.9-14.20130511.el7_4
  dependency: rtld(GNU_HASH)
    provider: glibc.x86_64 2.17-196.el7_4.2
    provider: glibc.i686 2.17-196.el7_4.2
...
    • only package names 
      $ yum deplist $(rpm -q gcc-gfortran) | awk '/provider:/ { print $2 }'|sort|uniq

bash.x86_64
gcc.x86_64
glibc.i686
glibc.x86_64
...

  • RPM what package provide libtinfo.so.5()(64bit)

    $ rpm -q --whatprovides 'libtinfo.so.5()(64bit)'

  ncurses-libs-5.9-14.20130511.el7_4.x86_64

  • YUM what package provide libtinfo.so.5()(64bit)

    $ yum whatprovides 'libtinfo.so.5()(64bit)'

ncurses-libs-5.9-13.20130511.el7.x86_64 : Ncurses libraries
Repo        : base
Matched from:
Provides    : libtinfo.so.5()(64bit)

ncurses-libs-5.9-14.20130511.el7_4.x86_64 : Ncurses libraries
Repo        : updates
Matched from:
Provides    : libtinfo.so.5()(64bit)

  • RPM package file integrity

    Package must be downloaded

$ rpm -vK gcc-gfortran-4.8.5-16.el7.x86_64.rpm

gcc-gfortran-4.8.5-16.el7.x86_64.rpm:
  Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
  Header SHA1 digest: OK (ae9044222237b6edb5c95e1dea593a7e99683596)
  V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
  MD5 digest: OK (15d89eebe4773f4c2e134636967901fb) 
  
NOKEY - means we do not have imported public key in RPM database
    • to verify package against signature, we need to:
      • download a package signature file
      • check the fingerprint
      • rpm --import the key

    Links:

  • Verifying an installed package

    • to verify if installed package is not modified

      $ rpm -V vim-minimal

empty if ok

    • if something missing e.g

      $ sudo rm -v /usr/bin/vi
removed ‘/usr/bin/vi’

rpm -V vim-minimal
missing     /usr/bin/vi

    • to fix inconsistency

      sudo yum reinstall vim-minimal

  • Force uninstall package with dependencies

    rpm -e --nodeps vim-common

--nodeps bypass the dependency checking

  • Downloading RPMs from repositories

    Will download only the requested package and its dependencies

$ yumdownloader --resolve gcc-gfortran.x86_64

after:

cpp-4.8.5-16.el7_4.2.x86_64.rpm
gcc-4.8.5-16.el7_4.2.x86_64.rpm
gcc-gfortran-4.8.5-16.el7_4.2.x86_64.rpm
...

  • Using rpm2cpio to unpack RPM package

    rpm2cpio ../gcc-gfortran-4.9.2-6.fc21.x86_64.rpm | cpio -idv

Links

103.1 Work on the command line

  • Bash shell

    • buildin commands, cd,break,exec
    • external commands
    • use 3 standard I/O streams
      • stdin
      • stdout
      • stderr
    • prompt for root mostly has #, but depend on configuration 
      [root@f28 ~]#

  • Commands and sequences

    • Echo 
      $ echo Hello
Hello

$ echo A sentence
A sentence

$ echo A   sentence
A sentence

$ echo "A   sentence"
A sentence

  • Escapes and line continuation

    $ echo -e "No new line\c"
No new line $
  
$ echo "A line with a typed
$ > return"
A line with a typed
> return

$ echo -e "A line with an escaped\nreturn"
A line with an escaped
return

$ echo "A line with an escaped\nreturn but no -e option"
A line with an escaped\nreturn but no -e option

$ echo -e Doubly escaped\\n\\tmetacharacters
Doubly escaped
      metacharacters

  • Bash shell metacharacters and control operators

    • metacharacters, | & ; ( ) < >
    • metacharacters must be either quoted or escaped using a backslash ()
    • control operators, || && & ; ;; | ( )
    • command sequences, non zero failure, zero success 
      $ echo line 1;echo line 2; echo line 3
line 1
line 2
line 3

$ echo line 1 && echo line 2 && echo line 3
line 1
line 2
line 3

$ echo line 1 || echo line 2; echo line 3
line 1
line 3

  • Exit

    $ (echo In subshell; exit 0) && echo OK  || echo Bad exit
In subshell
OK

$ (echo In subshell; exit 4) && echo OK  || echo Bad exit
In subshell
Bad exit

  • Environment variables

    • to reference shell variables prefix with $name
    • common bash environment variables 
      $ echo $PPID - show parent process PID

$ echo $$ - show current process PID, in this case shell process

$ echo $? - exit code of the last command
    • other e.g 
      $ (exit 0);echo $?;(exit 4);echo $?
0
4
    • variables are case sensitive so, var1 != VAR1 different variables
    • technically $$ and $? aren't shell variables, rather parameters, you can't assing value to them
    • create shell variable, without visibillity to child shell e.g 
      $ VAR1=var1
$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1

- it's empty
    • create shell variable, with visibillity to child shell e.g 
      $ export VAR1=var1

$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1
var1
    • shell variable from child shell process is not visible to parent shell e.g 
      $ export VAR1=var1

$ echo $VAR1
var1

$ ps -p $$ -o "pid ppid cmd"

PID  PPID CMD
2433  2406 -bash

$ bash

$ ps -p $$ -o "pid ppid cmd"

PID    PPID CMD
26905  2433 -bash

$ echo $VAR1
var1

- export VAR2 in child shell
  $ export VAR2=var2

  $ echo $VAR2
  var2

- exit from child shell
  
  $exit
  
  $ ps -p $$ -o "pid ppid cmd"

  PID  PPID CMD
  2433  2406 -bash
  
  $ echo $VAR2
  - it's empty
    • expansion double quotes(")/(single quotes(') e.g 
      $ echo "$SHELL" '$SHELL' "$$" '$$'
/bin/bash $SHELL 2597 $$

double quotes("), expansion is done before string si passed
$ bash -c "echo Expand in parent $$ $PPID"
Expand in parent 2597 2571

single quotes('), expansion is done after string si passed, so in child shell
$ bash -c 'echo expand in parent $$ $PPID'
Expand in parent 12109 2597
    • using curly braces with variable names 
      $ echo "-$HOME/abc-"
-/home/bodo/abc-

$ echo "-$HOME_abc-"
--

$ echo "-${HOME}_abc-"
-/home/bodo_abc-

  • Env

    • -i, start with an empty environment
    $ env
LANG=en_US.UTF-8
GDM_LANG=en_US.UTF-8
HISTCONTROL=ignoreboth
TODO=t
DISPLAY=:1
COLORTERM=truecolor
USERNAME=bodo
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins

$ env -i bash -c 'echo $SHELL; env'
/bin/bash
PWD=/home/bodo
SHLVL=1
_=/usr/bin/env
    • -i, start with an empty environment and -l login shell
    $ env -i bash -lc 'echo $SHELL; env'
/bin/bash
LANG=en_US.UTF-8
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins
PWD=/home/bodo
XDG_DATA_DIRS=/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

  • Unset and set

    • login shell
    • non-login shell
    • to display current shell options 
      $ echo $-
himBHs
    • set shell option 
      $ set -u
himuBHs
    • unset shell option 
      $ set +u
himBHs
    • e.g 
      $ echo $VAR1
var1

$unset VAR1;echo $VAR1
- empty

- causes bash to report an error with undefined variables rather than treat them as defined but empty
$ set -u;echo $VAR1
-bash: VAR1: unbound variable

  • Exec

    • replaces the current shell 
      $ echo $$
2597

$ bash
$ echo $$
8230

$ exec bash
$ echo $$
8230

$ exit

$ echo $$
2597

  • System information with uname

    Links:

  • Command history

    • history for commands on/off 
      On - this is default
$ set -o

Off
$ set +o
    • history command 
      $ history
3639  bash
3640  echo $$
3641  uname
3642  uname -s
3643  uname -n

- display last 2 commands
$ history 2
3660  man 4 tty
3661  history 2

- to delete command
$ history -d 3661
 3660  man 4 tty
 3661  history -d 3661
 3662  history 3
 
- run most recent command (also !-1)
$ !!
history 3
  3660  man 4 tty
  3661  history -d 3661
  3662  history 3

- with replacing a command parameter
$ env -i bash -c 'echo $$'
26085
$ !ec:s/$$/$PPID/
echo $PPID
2571
    • logout, the last $HISTSIZE line are saved to ~/.bash_history, multiple sessions could rewrite the .bash_history

  • PATHS

    • determined with $PATH variable
    • e.g how to find a command executable with which or type
    $ which bash env zip xclock echo set ls
/usr/bin/bash
/usr/bin/env
/usr/bin/zip
which: no xclock in (...
which: no set in (...
/usr/bin/ls

$ type bash env zip xclock echo set ls
bash is hashed (/usr/bin/bash)
env is hashed (/usr/bin/env)
zip is /usr/bin/zip
-bash: type: xclock: not found
echo is a shell builtin
set is a shell builtin
ls is aliased to `ls --color=auto'
  • absolute path /bin/echo
  • relative path 
    $ cd /usr

$ ../bin/echo Relative path
Relative path

  • Changing your working directory

    - display the path output when changed, in this example in /usr
export CDPATH=/usr:$CDPATH

$ cd /usr
$ cd local/
/usr/local/

- change to previous
$ cd -
/usr
$ cd -
/usr/local

- a previous path is stored in OLDPWD shell variable
$ echo $OLDPWD
/usr

  • Manual (man) pages

    • (1) User commands (ls,
    • (2) System calls or kernel function (link, mkdir)
    • (3) Library routines (btee, locale)
    • (4) Devices (tty)
    • (5) File format descriptions (keymaps, motd)
    • (6) Games
    • (7) Miscellaneous (arp,boot,regex)
    • (8) System administration (fsck, mount)
    - device
$ man 4 tty

- user command to print device name to stdout
$ man 1 tty

- man pages with name
$ whatis tty
tty (1)              - print the file name of the terminal connected to standard input
tty (1p)             - return user's terminal name
tty (4)              - controlling terminal

- apropos, search for keyword in man pages
$ apropos tty
__gnu_parallel::__binder1st (3) - Similar to std::binder1st, but giving the argument types explicitly
agetty (8)           - alternative Linux getty
CMS_get0_eContentType (3ssl) - get and set CMS content types and content
CMS_set1_eContentType (3ssl) - get and set CMS content types and content

103.3 Perform basic file and directory management

  • Listing directories

    • PWD shell variable

  • Listing details

    • On a storage device, a file or directory is contained in a collection of blocks
    • Information about a file is contained in an inode
    • inode number is also a file serial number it's unique whitina particular file system
    • ls doesn't list special files with (.) 
      -l --format=long

$ ls -la

-i list inodes numbers

$ ls -i

-d list directory itself not the contents

$ ls -ld

Sorting:
-t sorting by modification time
-S sorting by size

$ ls -lr
$ ls -lS

-r reverses

$ ls -ltr

    Links

  • Copying, moving, and deleting files

    • cp, copy
    • cp -p preserve attributes (modifi time, own ...etc) 
      $ cp -p text2 text2.bak_3

-rwxrwxr-x  1 bodo users    6 Mar 18 21:12 text2
-rwxrwxr-x  1 bodo users    6 Mar 18 21:12 text2.bak_3
    • mv, move/rename
    • rm, remove

  • Rmdir

    • you can remove only empty directories 
      mkdir -p dir2/dir3

$ rmdir dir2
rmdir: failed to remove 'dir2': Directory not empty

$rmdir dir2/dir3
$rmdir dir2

  • Recursive manipulation

    • copy recursively e.g. 
      cp -pR . copy1

  • Wildcards and globbing

    • man 7 glob
    ? - matches any single character
* - matches any string, including an empty string
[0-9] - character class

  • Touching files

    • touch updates modification time 
      $ touch f1

$ stat f1
modification time should change
    • touch with -c (or --no-create) 
      $ touch -c f3

$ ls -l f3
- empty
    • touch with timestamp -t 
      $ touch -t 201408121510.59 f3
-rw-r--r-- 1 bodo users 0 Aug 12  2014 f3
    • touch with datestring -d 
      touch -d 11am f4
-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4
    • touch with reference, means it uses date a timestamp of ref. file 
      -rw-r--r--  1 bodo users    0 Mar 19 11:00  f4

$ touch -r f4 f4a

$ ls -l f4*

-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4
-rw-r--r--  1 bodo users    0 Mar 19 11:00  f4a

  • Finding files

    • find with name 
      $ find . -name "*[1k]*"
./f1
./dir_1
./dir_1/new_text1
./text1
./text[*?!1]
./backup
./backup/text1.~2~
./backup/text1
./backup/text1.~1~
    • find with wholename 
      $ find . -wholename "*ack*1"
./backup/text1
    • find with ignorecase 
      $ find . -iwholename "*ACK*1"
./backup/text1
    • find a directories 
      $ find . -type d
.
./dir_1
./backup
    • find by specifics size (units, default 'b' 512 bytes block, 'c' bytes, 'k' kilobytes, 'M' megabytes) 
      - empty size files

-rw-r--r-- 1 bodo users 0 Aug 12  2014 f3a

$ find . -size 0
f3a

- find size between, ignore *back* pattern

-rw-r--r-- 2 bodo users   19 Mar 18 20:51  text1
-rw-r--r-- 1 bodo users    5 Mar 18 22:05 'text[*?!1]'
-rw-r--r-- 1 bodo users    6 Mar 18 21:12  text2
-rw-r--r-- 1 bodo users    5 Mar 18 22:05 'text[2*?!]'

$ find . ! -iwholename '*back*' -size +4c -size -20c -print
./text2
./text1
./text[*?!1]
./text[2*?!]
    • find with action -ls, equivalent to ls -lids 
      $ find . ! -iwholename '*back*' -size +4c -size -20c -ls
 18613131      4 -rw-r--r--   1  bodo     users           6 Mar 18 21:12 ./text2
 18613130      4 -rw-r--r--   2  bodo     users          19 Mar 18 20:51 ./text1
 18613144      4 -rw-r--r--   1  bodo     users           5 Mar 18 22:05 ./text[*?!1]
 18613145      4 -rw-r--r--   1  bodo     users           5 Mar 18 22:05 ./text[2*?!]
    • find with -exec 
      $ find . ! -iwholename '*back*' -size +4c -size -20c -exec ls -l '{}' \;
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 ./text2
-rw-r--r-- 2 bodo users 19 Mar 18 20:51 ./text1
-rw-r--r-- 1 bodo users 5 Mar 18 22:05 './text[*?!1]'
-rw-r--r-- 1 bodo users 5 Mar 18 22:05 './text[2*?!]'

- remove empty files
$ find . -empty -exec rm '{}' \;

- rename all *.htm files to *.html
$ find . -name "*.htm" -exec mv '{}' '{}l' \;
    • find within time range 
      -mtime -1 , less than 1*24
-mtime +1 , more than 1*24
- daystart, mean day start at 24:00, not relatively as mtime

      Link:

  • Identify files

    • /usr/bin/file 
      $ file /bin/echo
/bin/echo: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=f2aeec2985ecd663ef339313e6b0167722926e47, stripped

  • Compressing files

    • Using gzip and gunzip 
      Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ gzip text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.gz

Decompress:

$ gzip -d text2.gz

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2
    • Using xz and unxz 
      Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ xz text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.xz

Decompress:

$ unxz text2.xz

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2
    • Using bzip2 and bunzip2 
      Compress:

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2

$ bzip2 text2

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.bz2

Decompress:

$ bunzip -d text2.bz2

$ ls -l text2
-rw-r--r-- 1 bodo users 6 Mar 18 21:12 text2
    • compress uning stdout 
      gzip:

$ cat text2 | gzip -c > text2.gz

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.gz

xz:

$ xz - < text2 > text2.xz

$ ls -l text2*
-rw-r--r-- 1 bodo users 32 Mar 18 21:12 text2.xz

  • Using tar

    • diff archive 
      $ cd ../lpi103-2

$ ls -l
-rw-r--r--  1 bodo users    4 Mar 19 21:17  f1

$ tar -cvzf lpitar1.tar.gz .

$ ls -l lpitar1.tar.gz
-rw-r--r-- 1 bodo users 795 Mar 21 21:08 lpitar1.tar.gz

$ date
Wed Mar 21 21:21:04 CET 2018

$touch f1
-rw-r--r-- 1 bodo users 4 Mar 21 21:08 f1

$ tar --diff --file lpitar1.tar.gz .
./f1: Mod time differs
    • list archive with filtering 
      $ tar -tvf lpitar1.tar.gz --wildcards '*f*  
-rw-r--r-- bodo/users        0 2014-08-12 15:10 ./f3a
-rw-r--r-- bodo/users        0 2018-03-19 11:00 ./f4
-rw-r--r-- bodo/users        4 2018-03-19 21:17 ./f1
-rw-r--r-- bodo/users        0 2014-08-12 15:10 ./f3
-rw-r--r-- bodo/users        0 2018-03-19 11:00 ./f4b

  • Using cpio

103.4 Use streams, pipes, and redirects

  • Redirecting standard IO

    • n > - redirect to file descriptor, content lost without warnning

    • n >> - redirect to file descriptor, content append to end

      $ ls
text1  text2  text3  xaa  xab  yaa  yab

$ ls x* z*
ls: cannot access 'z*': No such file or directory
xaa   xab

$ ls x* z* >stdout.txt 2>stderr.txt

$ ls w* y*
ls: cannot access 'w*': No such file or directory
yaa   yab

$ ls w* y* >> stdout.txt 2>>stderr.txt

$ cat stdout.txt
xaa
xab
yaa
yab

$ cat stderr.txt
ls: cannot access 'z*': No such file or directory
ls: cannot access 'w*': No such file or directory

    • set -o noclobber, prevent overwriting of file (set off set +o noclobber)

      set -o noclobber

$ ls w* y* > stdout.txt 2>stderr.txt
-bash: stdout.txt: cannot overwrite existing file

Temporarily enable overwrite
$ ls w* y* >|stdout.txt 2>|stderr.txt

$ cat stdout.txt
yaa
yab

$ cat stderr.txt
ls: cannot access 'w*': No such file or directory

    • redirections, both 1 (stdout) and 2 (stderr)

      $ ls x* z* &>output.txt

    • redirections, both 1 (stdout) and 2 (stderr) explicit

      $ ls x* z* >output.txt 2>&1

      Links:

    • to /dev/null

      stderr to /dev/null

$ ls x* z* 2>/dev/null

  • Redirecting input

    • input redirection 
      $ cat text1
1 apple
2 pear
3 banana

$ tr ' ' '\t' < text1
1    apple
2    pear
3    banana
    • input redirection with here-document 
      $ sort -k2 <<END
> 1 apple
> 2 pear
> 3 banana
> END
1 apple
3 banana
2 pear
    • here-document with tabs 
      $ cat here-document.sh

cat <<- EOF
apple
EOF
    cat <<- EOF
    pear
EOF

$ . here-document.sh
apple
pear
    • here-document without tabs 
      $ cat here-document.sh

cat << EOF
apple
EOF
    cat << EOF
    pear
EOF

$ . here-document.sh
apple
    pear

  • Creating pipelines

    • piping stdout to stdin, when using hyphen(-) in place of filename it's indicating input should come from stdin rather than file 
      $ tar -tvf lpi103_4.tar.gz .

$ mkdir new

$ gzip -c -d lpi103_4.tar.gz | tar -xvf - -C new

$ ls new
ex-here.sh  stderr.txt  stdout.txt  text1 ...

  • Using output as arguments

    • xargs command 
      $ cat text1
1 apple
2 pear
3 banana

$ xargs < text1
1 apple 2 pear 3 banana

$ xargs < text1 echo "arg list:"
arg list: 1 apple 2 pear 3 banana

$ xargs --max-args 3 <text1 echo "arg list:"
arg list: 1 apple 2
arg list: pear 3 banana

$ xargs -n 1 <text1 echo "arg list:"
arg list: 1
arg list: apple
arg list: 2
arg list: pear
arg list: 3
arg list: banana
    • using xargs with quoting 
      $ echo '"4 plums"' | cat text1 - | xargs -n 1
1
apple
2
pear
3
banana
4 plums
    • using xargs with lines of input 
      Replace:

$ xargs -I XYZ echo "START XYZ REPEAT XYZ END" <text1
START 1 apple REPEAT 1 apple END
START 2 pear REPEAT 2 pear END
START 3 banana REPEAT 3 banana END

$ xargs -IX echo "<X><X>" <text2
<9      plum><9 plum>
<3      banana><3       banana>
<10     apple><10       apple>

$ cat text1 text2 | xargs -L2
1 apple 2 pear
3 banana 9 plum
3 banana 10 apple
    • using xargs with lists of files 
      $ ls |xargs grep "1"
text1:1 apple
text2:10    apple
xaa:1 apple
yaa:1
    • show limits, in old kernels xargs max. number of argumets was limited 
      $ xargs --show-limits
Your environment variables take up 1691 bytes
POSIX upper limit on argument length (this system): 2093413

  • Using the find command with the -exec option or with xargs

    With exec (calling for each found file, so so text3 -> text1 -> text3 -> text2):

$ find text[12] -exec cat text3 "{}" \;
This is a sentence.  This is a sentence.  This is a sentence. 
1 apple
2 pear
3 banana
This is a sentence.  This is a sentence.  This is a sentence. 
9       plum
3       banana
10      apple

With xargs (calling all together, so text3 -> text1 -> text2):

$ find text[12] | xargs -t cat text3
This is a sentence.  This is a sentence.  This is a sentence. 
1 apple
2 pear
3 banana
9       plum
3       banana
10      apple
  • using find and xargs with blanks in file names 
    There is "text 1" copy.  
-print0 option with find if you are piping the output to xargs, 
 and use the -0 option to tell xargs to expect null-delimited input.

$ find . -name "*1" -print0 |xargs -0 grep "1"
./text1:1 apple
./text 1:1 apple

same as,

$ grep "1" "text 1" text1
./text1:1 apple
./text 1:1 apple

    Links

  • Splitting output
    • tee command, duplicate the output to file
    $ ls text[1-3] | tee f1 f2

$ cat f1
text1
text2
text3

$ cat f2
text1
text2
text3

103.5 Create, monitor, and kill processes

  • Manage foreground and background jobs

    • process.sh script 
      process.sh

#!/bin/bash

echo "Begin of count process"

for i in $(seq 1 1000); do
     echo "i: $i"
     sleep 5
done

echo "End of count process"

exit 0
    • fg command 
      $ ./process.sh
i: 1
i: 2
...

To stop process

Ctrl+Z

[1]+  Stopped                 ./process.sh

$ jobs -l
[1]+  3005 Stopped                 ./process.sh

Bring process foreground again

$ fg %1
i: 3
...

Stop again

Ctrl+Z

[1]+  Stopped                 ./process.sh

$ fg %?pro
i: 4
...
    • put process to backgroud 
      $ ./process.sh
i: 1
i: 2
...

To stop process

Ctrl+Z

[1]+  Stopped                 ./process.sh

To background:

$ bg

To stop again

$ fg

Ctrl+Z

[1]+  Stopped                 ./process.sh

  • Standard IO and background processes

    • if stdin is requested and run in background, then bash will stop the process 
      $ (date; cat - > bginput.txt; date) &
[1] 19728
Sat Mar 24 20:19:28 CET 2018

[1]+  Stopped                 ( date; cat - > bginput.txt; date )

$ fg
( date; cat - > bginput.txt; date )
ok
and back

Ctrl + D

$ cat bginput.txt
ok
and back

  • Run a process after log out

    • nohup 
      $ nohup ./process.sh 1 > job_1.txt 2>&1 &
[1] 12646

  • Monitor process

    • ps command 
      $ nohup ./process.sh 1 > job_1.txt 2>&1 &
$ nohup ./process.sh 1 > job_1.txt 2>&1 &
$ nohup ./process.sh 1 > job_1.txt 2>&1 &

[2]  16124 Running                 nohup ./process.sh 1 > job_1.txt 2>&1 &
[3]- 16220 Running                 nohup ./process.sh 2 > job_1.txt 2>&1 &
[4]+ 16316 Running                 nohup ./process.sh 3 > job_1.txt 2>&1 &

$ jobs -p
16124
16220
16316

$ jobs -p | xargs ps -F -p
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
bodo     17102  2074  0  3494  3388   2 21:14 pts/1    S      0:00 /bin/bash ./process.sh 1
bodo     17164  2074  0  3494  3376   0 21:14 pts/1    S      0:00 /bin/bash ./process.sh 2
bodo     17227  2074  0  3494  3336   6 21:14 pts/1    S      0:00 /bin/bash ./process.sh 3
    • to display like tree like structure ps -j --forest 
      $ ps -j --forest

PID  PGID   SID TTY          TIME CMD
2074  2074  2074 pts/1    00:00:00 bash
23013 23013  2074 pts/1    00:00:00  \_ ps
    • free displays the amount of free and used memory in your system 
      Displays -t total memory in -g gigabytes
$ free -tg

Displays -t total memory in -m gigabytes every 10 seconds
$ free -tms 10

  • Select and sort processes for display

    • ps to display all processes with controlling terminals (-a all, -f full) 
      $ ps -af
    • ps to display all processes without controlling terminals 
      $ ps -xf
    • ps display for particular command and custom output (-C process command, -o customized output) 
      $ ps -C bash -o user,pid,tty,time,comm
    • ps sort with order (+) ascending, (-) descending 
      sid - process group session id

$ ps -aj --sort +sid;-comm
    • pgrep display process defined as regex 
      $ pgrep -af system*
    • top - interactive ps

Links

  • Send signals to processes

    • list of Signals 
      $ kill -l

man 7 signal
    • send stop signal to job (same as sent by Ctrl + Z) 
      $ nohup ./process.sh 1 > job.txt 2>&1 &
[1] 4976

$ kill -s SIGSTOP %1
[1]+  Stopped                 nohup ./process.sh > job_1.txt 2>&1
    • send continue signal to job 
      $ kill -s SIGCONT %1
    • sent terminatig signal 
      SIGTERM - 15 

$ kill -s SIGTERM %1
[1]+  Terminated              nohup ./process.sh > job_1.txt 2>&1

Alternative:

$ kill -15 %1
[1]+  Terminated              nohup ./process.sh > job_1.txt 2>&1
    • terminating process unconditionally SIGKILL 
      $  nohup ./process.sh > job_1.txt 2>&1 &
[1] 20320

$ kill -9 %1
[1]+  Killed                  nohup ./process.sh > job_1.txt 2>&1

    Links:

  • Logout and nohup

    • after logout nohup processes continued to work but they are detached from terminal 
      In pts1:

$  nohup ./process.sh > job_1.txt 2>&1 &
[1] 20320

$ exit

Open new terminal:

$ jobs -l
- empty

$ ps -C process.sh
PID TTY          TIME CMD
31715 ?        00:00:00 process.sh

? - not attached to any terminal

103.6 Modify process execution priorities

  • Using ps to find niceness 
    $ ps -l

Link

104.1 Create partitions and filesystems

  • MBR Partitions

    • limited to 4 primary partitions (1-4)
    • logical partitions >= 5 (gaps are not allowed)
    • backup MBR e.g., sfdisk -d /dev/sda > sda-backup.txt
    • restore MBR e.g., sfdisk -d /dev/sda < sda-backup.txt
    • MBR partitions type code:
      • 0x0c(FAT)
      • 0x05(old type of extended partitions)
      • 0x07(NTFS)
      • 0x0f(a newer type of extended partitions)
      • 0x82(Linux swap)
      • 0x83(Linux filesystem)

  • GPT Partitions

    • can define up to 128
    • gaps can occure, e.g. three partitions numbered 3,9,55
    • GPT supports partition type codes, it's 16-byte GUID (128 bits) GUID

  • LVM - logical volume management

    • access the logical volumes /dev/mapper (e.g. /dev/mapper/myvol-home)
    • /boot cannot reside in LVM when using a boot loader which does not support LVM (GRUB 2 only support it)

    Links

104.3 Control mounting and unmounting of filesystems

  • Mounting filesystems

    • mount the filesystem on the device
    • mounting a device requires root permissions or sudo
    • mount point must exist before you mount anything over it
    • if mount point contains subdirectories, after mount they are not visible
    • to list what is mounted use mount without an parameters, similar info /proc/mounts, /etc/mtab
    • mounting a specific filesystem type mount -v vfat /dev/sda3 /dos

  • Mount options

    • mount options override default behaviour
    • remount already mounted filesystem read-only 
      mount -o remount,ro /dos
    • remount already mounted filesystem read-write 
      mount -o remount,rw /dos
    • remount will not complete successfully if any process has open files or directories in the filesystem you are remounting

  • Labels, UUIDs, and links

    • to find UUID and Label(if present) of device 
      /dev/sdc - usb key

# blkid /dev/sdc1
/dev/sdc1: LABEL="ARCH_201508" UUID="FC75-EDAF" TYPE="vfat" PARTUUID="052cf3ed-01"

# mkdir -p /mnt/ARCH_201508

To mount with LABEL
# mount LABEL="ARCH_201508" /mnt/ARCH_201508
# ls -l /mnt/ARCH_201508
arch
...

# umount /mnt/ARCH_201508

To mount with UUID
# mount UUID="FC75-EDAF" /mnt/ARCH_201508
# ls -l /mnt/ARCH_201508
arch
    • to find symbolic link to /dev/sdc1 in /dev tree 
      $ find /dev -lname "*sdc1"
/dev/disk/by-label/ARCH_201508
/dev/disk/by-uuid/FC75-EDAF
/dev/disk/by-partuuid/052cf3ed-01
/dev/disk/by-path/pci-0000:00:14.0-usb-0:4:1.0-scsi-0:0:0:0-part1
/dev/disk/by-id/usb-Kingston_DataTraveler_108_001A4D5F1A5CBB5130000199-0:0-part1
/dev/block/8:33

  • Boot time and fstab

    • root= parameter in GRUB and LILO to tell the boot loader what filesystem should be mounted as root

    • set root in GRUB2

    • after mounting, initialization run mount -a to automatically mount set of filesystems defined in /etc/fstab

      $ cat /etc/fstab
# /dev/sda1
UUID=1638d092-0700-4688-b342-ea3051f9ffa2       /               ext4            defaults,noatime,discard        0 1

    • fstab format:

      • file sysytem, LABEL, UUID
      • mount point
      • type, should aldo by auto, ext4, btrfs .etc
      • option:
        • ro, read only
        • rw, reda-write
        • noauto, not automatically mounted at boot time
        • user, non-root user is permitted to mount/unmout media, now part of udev
        • exec, to allow execute files from mounted system
        • noexec, to not allow execute files from mounted system
        • noatime, do not reccord access times
      • dump, consider ext2 a ext3 for backups filesystem 0 - off, 1 - on
      • pass, pass the order to check filesystem at boot time 0 - off, 1 on for root partition, 2 on for non root partition

      Links

  • Unmounting filesystems

    • it isn't possible to umount if there are processes with opend files from that filesystem
    • lsof 
      USB is mounted in /mnt/ARCH_201508/

# umount /mnt/ARCH_201508/
umount: /mnt/ARCH_201508/: target is busy

# lsof +f -- /mnt/ARCH_201508/ | grep -i --color DIR
bash      11913 root  cwd    DIR    8,33     4096     1 /mnt/ARCH_201508
vim       14361 root  cwd    DIR    8,33     4096     1 /mnt/ARCH_201508

# fuser -uvm /mnt/ARCH_201508/
                     USER     PID    ACCESS COMMAND
/mnt/ARCH_201508:    root     kernel mount  (root)/run/media/bodo/ARCH_201508
                     root      11913 ..c..  (root)bash
                     root      14361 ..c..  (root)vim

# umount /mnt/ARCH_201508/
- should successfully umounted

  • Removable filesystems

    • to enable u/mount under user specify 'uid=1000'

  • Swap space

    • to display swap device use swapon -s or cat /proc/swaps
    • to customize output 
      $ swapon --show=NAME,TYPE,SIZE
NAME      TYPE      SIZE
/dev/sda3 partition  12G

    Links

104.5 Manage file permissions and ownership

  • User and groups

    • identify user currently logged whoami
    • identify user groups 
      $ id
uid=1000(bodo) gid=100(users) groups=100(users)

$ groups
users

  • File ownership and permissions

    • (r) read, (w) write, (x) execute
    • directories:
      • (r) enable list content of directory
      • (w) create or delete files in the directory
      • (x) allow enter directory and access subdirectories
    • create files in subdirectories, "root" 0400 (dr--------), subdirectory (0700) (drwx------) 
      "root"       - dr-------- 3 bodo users 4096 Mar 25 17:02 dir1
                                                         |_  
subdirectory - drwx------ 2 bodo users 4096 Mar 25 18:53 subdir1

$ touch dir1/subdir1/subdir1_script.sh
touch: cannot touch 'dir1/subdir1/subdir1_script.sh': Permission denied
    • create files in subdirectories, "root" 0200 (d-w-------), subdirectory (0700) (drwx------) 
      $ touch dir1/subdir1/subdir1_script.sh
touch: cannot touch 'dir1/subdir1/subdir1_script.sh': Permission denied
    • create files in subdirectories, "root" 0100 (d--x------), subdirectory (0700) (drwx------) 
      $ touch dir1/subdir1/subdir1_script.sh

$ ls -l dir1/subdir1/subdir1_script.sh
-rw-r--r-- 1 bodo users 0 Mar 25 19:05 dir1/subdir1/subdir1_script.sh

  • Other filesystem objects

    • (-) regular file
    • d directory
    • l symbolic link
    • c character special device
    • b block special device
    • p FIFO pipe
    • s Socket

  • The eleventh character

    • recent enhacenment, not in all distributions

  • Changing permissions

    • adding permissions to all 
      $ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh

$ chmod -v +x script.sh
mode of 'script.sh' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x)
    • adding premissions selective 
      $ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh

$ chmod -v o+x script.sh
mode of 'script.sh' changed from 0644 (rw-r--r--) to 0645 (rw-r--r-x)

$ chmod -v g+x script.sh
mode of 'script.sh' changed from 0645 (rw-r--r-x) to 0655 (rw-r-xr-x)

$ chmod -v u+x script.sh
mode of 'script.sh' changed from 0655 (rw-r-xr-x) to 0755 (rwxr-xr-x)

Same:

$ chmod -v ugo+x script.sh
mode of 'script.sh' changed from 0655 (rw-r-xr-x) to 0755 (rwxr-xr-x)
    • remove premissions selective 
      $ chmod -v ugo-x script.sh

$ ls -l script.sh
-rw-r--r-- 1 bodo users    0 Mar 25 19:18 script.sh

  • Octal permissions

    • 0 ---
    • 1 --x
    • 2 -w-
    • 3 -wx
    • 4 r--
    • 5 r-x
    • 6 rw-
    • 7 rwx

  • Access modes

    • suid, When an executable program has the suid access modes set, it will run as if it had been started by the file's owner, rather than by the user who really started it. 
      It's (s)

-rwsr-xr-x 1 root root 55416 Dec 16 17:03 /usr/bin/passwd
    • set a remove suid 
      Set:

$ chmod u+s script.sh
-rwsr-xrwx 1 bodo users    0 Mar 25 19:18 script.sh

Remove:
chmod u-s script.sh
    • set a remove guid 
      Set:

chmod g+s script.sh
-rwxr-srwx 1 bodo users    0 Mar 25 19:18 script.sh

Remove:
chmod g-s script.sh
    • Directory with sgid any files or directories created in it will inherit the group ID of the directory
    • Sticky bit, if set only user or root can delete or unlink the file 
      $ chmod -v +t dir1
mode of 'dir1' changed from 0755 (rwxr-xr-x) to 1755 (rwxr-xr-t)

  • Immutable files

    As root:

$ chattr +i /share/temp/lpi/lpi104-5/script.sh

$lsattr /share/temp/lpi/lpi104-5/script.sh
----i---------e--- /share/temp/lpi/lpi104-5/script.sh

  • The file creation mask

    • represents a numeric permission value to be removed
    $ umask
022 (that is u=rwx,g=rx,o=rx)

In this case group, owner write permission

    Links

104.6 Create and change hard and symbolic links

  • Hardlinks
  • Softlinks
  • Identifying links
  • Who links to me?
    • finding hardlinks 
      $ find lpi104-6 -samefile lpi104-6/file1
lpi104-6/file3
lpi104-6/file1
lpi104-6/subdir/file3sub

$ ls -i lpi104-6/
18744944 file1

$ find lpi104-6 -inum 18744944
lpi104-6/file3
lpi104-6/file1
lpi104-6/subdir/file3sub
    • finding softlinks 
      $ find lpi104-6 -lname "*file1"
lpi104-6/file4
lpi104-6/file6

Resources:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment