Shell scripts and one-liners

Shell scripts and one liners

Snippets, tips and tricks in no particular order. FOR ADVANCED USERS ONLY. Some of these one liners may be dangerous since they might delete files in loops. Please pay attention and make backups. :-)

rsync examples

Usage:

rsync --verbose --archive --recursive --checksum --prune-empty-dirs --stats --human-readable --safe-links source-path user@server:/destination-path/

-v, --verbose verbose
-a, --archive archive mode (preserves permissions, links etc...)
-r, --recursive recursive
-c, --checksum skip based on checksum, not mod-time & size
n or --dry-run trial run, no changes
--safe-links
--stats show statistics
--human-readable
z, --compress compress file data during the transfer
-m, --prune-empty-dirs
-e, --rsh run command on remote shell
--delete delete extraneous files from dest dirs

Use the last uption for making exact mirros.

Check hard disks

df
more /etc/mtab
ls /sys/block/* | grep block | grep sd | wc -l

word frequency

word frequency thanks to this post

tr '[A-Z]' '[a-z]' < file.txt | tr -cd '[A-Za-z0-9_ \012]' | tr -s '[ ]' '\012' | sort | uniq -c | sort -nr

tar archives

make archive:

tar czvf blah.tar.gz *.txt

list the CRC checksum:

gzip -lv blah.tar.gz

list files inside (also for test):

tar -tvf blah.tar.gz

add one file to it:

gunzip blah.tar.gz
tar -zf blah.tar.gz new_file
gzip blah.tar.gz

get only java files out of archive:

tar zxvf blah.tar.gz --wildcards --no-anchored '*.java'

test tar.gz

gunzip -tv blah.tar.gz

tar over SSH:

tar cvzf - /wwwdata | ssh user@server "dd of=/backup/wwwdata.tar.gz"
tar zcvf - /wwwdata | ssh user@server "cat &gt; /backup/wwwdata.tar.gz"
ssh [email protected] "cat /backup/wwwdata.tar.gz" | tar zxvf -

Compare line count

 if [[ $(cat file2.ext | wc -l) -eq $(cat file2.ext | wc -l) ]];then echo 'Equal';else echo 'Not equal';fi

Cut video with ffmpeg

Type:

ffmpeg -ss 00:00:30 -i inputfile -t 00:00:05 -vcodec copy -acodec copy outputfile

GNU Screen survival

Start a SSH tunnel session with GNU Screen:

ssh -t user@server screen -D -R

or on a Raspberry Pi:

ssh -t [email protected] "TERM=xterm /opt/bin/screen" -D -R

Some commands:

^A ^W - window list, where am I
^A ^C - create new window
^A space - next window
^A p - previous window
^A ^A - switch to previous screen (toggle)
^A [0-9] - go to window [0-9]
^A esc - copy mode, which I use for scroll back
^A ^K - kill a window (better to use ctrl-D or exit)
^Q and ^A ^Q to try to unlock it.
^A ? - show the help screen!
^A two times a normal ^A meaning beginning of the line
^A d - detach

Determine host name

host SERVER_ADDRESS | ip2host

Make animated GIF

convert -delay 10 -loop 0 *.png animation.gif

macports

some commands:

port echo leaves
sudo port uninstall leaves
sudo port uninstall inactive
sudo port uninstall --follow-dependencies portname

sudo port upgrade outdated
sudo port selfupdate
sudo port -d selfupdate

Convert DAT to MPG

for i in *.DAT; do o=`basename $i .DAT`;vcdgear -dat2mpg $i ~/$o.mpg; done

Updating owncloud via FTP

Make backup of everything.
Delete all folders except config and data
unpack the owncloud tarball on pc
edit the .htaccess file from tarball to add a # in front of the line # Options -Indexes
upload everything to the server via FTP and take care of the permissions

Apple Mail attachment problem

To change the default behaviour of the attachments in the mail program:

defaults write com.apple.mail DisableInlineAttachmentViewing -bool yes

to set back to the default behavior:

defaults write com.apple.mail DisableInlineAttachmentViewing -bool false

Insert a line at the beginning of a file

Thanks to this post it is easy to insert a line at the begining of file.

#!/bin/bash
# script insertline
# insert a line at the beginning of an ascii file
TXT="#temperature(C)    0.0"
(echo "$TXT"; cat $1) > $1.new
mv $1{.new,}

Finally use it with the following command:

# find ./ -iname input.file.txt -exec ./insertline {} \;

Permutation of strings

After a long day of work, you type everything else than "make" or other commands that you use very often. In order to help yourself, just put the permutation in the BASH profile like this:

ruby -e '"make".chars.permutation.map(&amp;:join).each {|str| puts "alias "+str.to_s+"=\"make\""}' | cat &gt;&gt; .bash_profile

Change dates in EXIF Tool

find ./ -iname "*.jpg" -exec exiftool "-AllDates=2013-08-27 12:00:00" {} \;

OwnCloud UI doesn't delete files?

Just check your control panel and find the configuration section and set the following PHP value to off

magic_quotes_gpc Off

File with random content

Define block size and count:

dd if=/dev/urandom of=ran.dom bs=8 count=1

Mathematica fonts error over remote X session

goto http://support.wolfram.com/
search for MathFonts and download tgz file to your local computer and unpack it.
under debian create folder Mathematica in:
```
  /usr/share/fonts/Mathematica
```
under Mac OS X, assuming Macports installation create the folder in:
```
  /opt/local/share/fonts/Mathematica
```
then copy BDF and Type1 folders there.

then use "sux" (or sudo with graphics)

  xset fp+ /usr/share/fonts/Mathematica/BDF
  xset fp+ /usr/share/fonts/Mathematica/Type1

  xset fp+ /opt/local/share/fonts/Mathematica/BDF
  xset fp+ /opt/local/share/fonts/Mathematica/Type1

under Mac OS X.

Then ssh -X to the remote computer where Mathematica is installed.
run mathematica.

OSX: compile cern ROOT

I was building svn root under mac, compilation was successful but the linker had a problem:

ld: in /opt/local/lib, can't map file, errno=22

After searching, it seemed it is a common problem. Then I opened the Makefile.config and searched for /opt/local/lib one of them FTGLLIBDIR was without -L. After fixing it compiled fine.

OSX: Poster with Keynote

Presentation using Keynote:

Paper size: A0 Paper mm: 841 × 1189 inch: 33.1 × 46.8 Pixel: 2383.2 x 3369.6 for Keynote every inch is 72 DPI. change to percent notation first corner distance: 1.5% from top 2.5% from left

Uninstall an application under OSX

Warning! the following code could DAMAGE your mac os x installation if improperly used. So be careful and use it only if you know what you do.

lsbom -f -l -s -pf Archive.bom | awk '{print "rm -rvf " $1}' &gt; remover

then make remover executable. The rest you guess yourself.

extract data from the .C ROOT file

sed 's|[^(]*(\([^)]*\)).*|\1|' 103379210_c_projection_1001.C &gt; newfile.txt

Process multiple files in a for loop

one example here:

time for i in *.DAT;do vcdgear -dat2mpg  $i ~/Desktop/`basename $i .DAT`.mpg;done

another example:

time for i in *.dv;do ffmpeg -i $i -sameq -deterlace `basename $i .dv`.avi;done

here a multiliner:

#!/bin/bash
#
# This script searches all the subdirectories for Flash Videos (e.g. YouTube)
# and extracts the MP3 Audio out of them.and saves them as an MP3 file near the original FLV.
#

echo "Searching in current and all subdirectories for Flash Video (*.flv) files ...";
LIST=`find ./ -iname '*.[fF][lL][vV]' | sed -e 's/\ /IHopeThisExpressionNeverOccursInAFileName/g'`;
#it is 3:45 A.M. Sorry for this awful technique!
for d in $LIST; do
    file=`echo $d | sed -e 's/IHopeThisExpressionNeverOccursInAFileName/\ /g'`;
    ffmpeg -sameq -i "$file" "${file%*.flv}.mp3";
done

another:

#!/bin/bash
for i in *.flv; do ffmpeg -i $i -f mp3 -vn -acodec copy `basename $i .flv`.mp3;done

an exmaple with mencoder:

mencoder MVI_3192.AVI -ovc xvid -oac mp3lame -xvidencopts bitrate=-10000 -o new.avi

or lame:

#!/bin/bash
for i in *.AVI; do mencoder $i -ovc xvid -oac mp3lame -xvidencopts bitrate=-10000 -o `basename $i _small.avi`.avi;done

SSH without login

a@A:~&gt; ssh-keygen -t rsa
a@A:~&gt; ssh b@B mkdir -p .ssh
a@A:~&gt; cat .ssh/id_rsa.pub | ssh b@B 'cat &gt;&gt; .ssh/authorized_keys'
a@A:~&gt; ssh b@B hostname
B

CP without overwrite

Sometimes a long copy process is interrupted. In order to continue one must avoid overwrite existing files. Use / at the end to take the content, not the whole directory.

cp -vrn /source/src1/ /dest/dest1

SED delete first line from file

sed '1d' file.txt

AWK print selected columns

Delete second to last columns of a data file. Change numbers for other ranges:

awk '{for(i=2;i

after %s you can choose another delimiter. Taken from:

SED search and replace

sed again :-) be careful of the overwrite option -i should not be used on large files, when disc is exhausted could cause data loss.

sed -i "" 's/;/ /g' input.file

This means search for semicolon in all lines of the input.file, replace it with one space character and do it for all other occurrences in that line (option g).

Command line Email using MUTT

Before starting, setup your .muttrc file:

echo 'set from="real name <adress@mail>"' >> .muttrc

then

echo 'Body message' | mutt -s 'Subject here' -a file.one file.two file.three -c [email protected] -b [email protected] -- adr1@mail adr2@mail ...

SED Example

remove lines that start with "SOMETHING".

sed -e '/^SOMETHING/d' inputfile &gt; outputfile

Converting PS to JPG

Use this one liner:

for i in *.ps; do o=`basename $i .ps`;convert -verbose $i $o.jpg; done

Converting AVI to MPEG

Use ffmpeg like:

ffmpeg -i movie.AVI -sameq movie.mpg

Xorg Errors

To see what problems xorg might have had:

cat /var/log/Xorg.0.log | grep EE

Formatting SD Card or USB Stick

Make sure the device is not mounted. Use 'mkfs.vfat' which is the same as 'mkdosfs':

mkdosfs -I /dev/sdd

XRANDR

Configuration for xrandr: With 'Xrandr' you can dynamically change the resolution of the display:

xrandr --output VGA --off --output LVDS --mode 1280x800

or with:

xrandr --output VGA --mode 1024x768 --output LVDS --off

DCOP

Some programs use DCOP command structure, for example:

dcop amarok player play

LDD

with 'ldd' you can see the dynamically linked libraries in memory e.g.:

ldd qana
ps auxwww| grep qana
cat /proc/"PROCESSNUMMER"/maps

Converting FLV to MP3

Using the following script you can convert FLV Flash Videos to MP3. Also many thanks to .

#!/bin/bash
#
# This script searches all the subdirectories for Flash Videos (e.g. YouTube)
# and extracts the MP3 Audio out of them.and saves them as an MP3 file near the original FLV.
#

echo "Searching in current and all subdirectories for Flash Video (*.flv) files ...";
LIST=`find ./ -iname '*.[fF][lL][vV]' | sed -e 's/\ /IHopeThisExpressionNeverOccursInAFileName/g'`;
#it is 3:45 A.M. Sorry for this awful technique!
for d in $LIST; do
    file=`echo $d | sed -e 's/IHopeThisExpressionNeverOccursInAFileName/\ /g'`;
    ffmpeg -i "$file" -y -acodec copy "${file%*.flv}.mp3";
done

Recover Crashed Terminal

Some programs crash the font on the terminal console. Do as follows:

c-v c-o

Simple shell script

This is a simple bash script which traverses the directories and does the specified command on the file.

#!/bin/bash
echo $@
echo $1
echo $2

dummy=`find $1 -iname *.$2`
# dummy=`find $@ -iname '*html'`
# for d in *.html; do
for d in $dummy; do
echo $d;
echo ${d%*.$2};
# echo ${d#*.$2};
done

EMACS: compile lisp code

emacs -batch -q -f batch-byte-compile file.el

Conver Flash Video to MP3

To extract the MP3 from the Flash Video try this:

ffmpeg -i input_file.flv -f mp3 -vn -acodec copy output_file.mp3

This will extract an MP3 which is compatible with K3B.

Convert windows-1256 to utf-8

You need iconv installed. It is under /usr/ports/converters/iconv in FreeBSD. Then you type:

iconv -f windows-1256 -t utf-8 old.html &gt; new.html

This also works for XML or other documents. For a list of supported encodings on your platform type:

iconv -l

For bigger documents (> 1GB) it may be better to use siconv which is also available in the ports collection /usr/ports/converters/siconv in the FreeBSD distribution. You can use it like this

cat old.html | siconv windows-1256 utf-8 &gt; new.html

This works like a real charm :)

xterm with UTF support

xterm -class UXTerm -u8 -fg white -bg black -fn -misc-fixed-medium-r-normal--18-120-100-100-c-90-iso1064

Display with SU-Xterm

sometimes when you write 'xterm -s' and start a root xterm, you may not have the display parameter set. so do:

root# xhost +localhost
root# export DISPLAY=":0.0"

mounting an iso file:

Mounting:

vnconfig -c /dev/vn0 image.iso
mount -t cd9660 /dev/vn0c /mnt/whatever

Unmounting:

umount /mnt/whatever
vnconfig -u /dev/vn0

find and simultaneous chmod of directories

An example:

user$ find ./Personal-Docs/ -type d -exec chmod 755 {} \;
user$ find ./Personal-Docs/ -type f -exec chmod 644 {} \;
user$ find /usr/ports/comms/ -type f -name pkg-descr -exec cat {} \; | more
user$ find ./* -name '*html' -type f -exec sed -i "" -e 's/July 2004/January 2005/g' {} \;
user$ find ./* -type f -exec sed -i "" -e 's/.$//' {} \;
user$ find ./ -type f -iname  '*.core' -exec rm -v {} \;

cvs update:

cvsup -L 2 -g /etc/ports-supfile
cd /usr/ports
make fetchindex
portsdb -u

Running a jar file:

sudo java -noverify -cp rfid.jar de.tudarmstadt.ito.rfid.reader.ReaderManager -c ./

sub partitions under FBSD 4.9

a : 327680
b :      1
c:      82
d:      83
         4
mknod ad0s4a c 116 327680 root:operator
mount -o rw -f /dev/ad0s4a /mnt

XMODMap

setxkbmap -layout bg -variant phonetic -option "grp:alt_shift_toggle" -compat "g
roup_led"

setxkbmap -layout gb -option "grp:alt_shift_toggle" -compat "group_led"

A neat PostScript rendering of a manual page

use ``-t'' switch
of the man(1) utility: ``man -t ''.  For example:

        man -t grep &gt; grep.ps   # Save the PostScript version to a file
or
        man -t printf | lp      # Send the PostScript directly to printer

Remove all those ^M characters from a DOS file

col -bx  newfile

add user to the specified group.

pw groupmod dialer -m xara

SSH Key generation

public and private:

ssh-keygen -t dsa

The generated key will be in '.ssh/id_dsa.pub' to change an existing password:

ssh-keygen -f ./id_dsa  -p

Diff and Patch

diff -bur fontutils.c.orig fontutils.c &gt; blah.diff
patch -p0

Burning Data CDs under FreeBSD

best direct burning method:

mkisofs -rU -V Pharmakobotanik  /mnt/xara/Pharmako/ | burncd -emv -s 2 -f /dev/acd0c data  - fixate

one can let the -r away.

mkisofs -v -U -V label -o /usr/temp/file.iso /usr/dir
burncd -emv -s 2 -f /dev/acd0 data /usr/temp/file.iso fixate

you can add -J to add juliett file extensions for WDose.

Copying data CDs directly

assuming that you have two CD-ROM drives on your computer, and the writer is '/dev/acd0' you can copy CDs like this:

dd if=/dev/acd1 bs=2048 | burncd -ev -s 24 -f /dev/acd0 data - fixate

Burning AudioCD from MP3s under FreeBSD:

Here is how to do it:

mpg123 -w - anysong.mp3 &gt; newsong.wav
sox -t wav -r 44100 -s -w -c 2 track.wav track.raw
burncd -f /dev/acd0c -s 8 audio *.raw fixate

Plotting in GNUPLOT

plot "data2" using 1:2

Proxy

Setting up proxy in the shell is easy:

export FTP_PROXY="http://proxy.tu-darmstadt.de:80"
export HTTP_PROXY="http://proxy.tu-darmstadt.de:80"

make master boot record writable

sysctl -a | grep geom

so that you know the name. then:

sysctl kern.geom.debugflags=16
boot0cfg

Now you ask why on earth should you make master boot record writable??!!

To prevent SSH-D to delay at startup

change

UseDNS no

in /etc/ssh/sshd_config

SourceForge CVS Check out

To check out a module name just type the directory name without path.

cvs -d:pserver:[email protected]:/cvsroot/pcb login
cvs -z3 -d:pserver:[email protected]:/cvsroot/pcb co -P pcb

Press ENTER for password.

Umlaut characters in Emacs

to print german umlaut in Emacs:

[c-x 8 "]

then u,o or a

Install path

You want to know where your port is installed? Acquire the complete port name with 'pkg_info' then use:

less /var/db/pkg/port_name_comes_here/+CONTENTS

Xdefaults changes

If you want the changes in .Xdefaults to take effect without a restart, then run:

xrdb -merge .Xdefaults

Some commands

strace follows the system calls. tail -f follows the last parts of a file as it is being written on. xrander -s changes the resolution of the screen sudo nmap -sS scans the ports of an DUT computer (like an old AIX computer!)

See who is using the sound card!

lsof -w $( find /dev -group audio )

Using Timidity

Type:

timidity -Ov1S *.mid

If you want .wav files instead, change the v to a w.

Misc commands

Here are some more commands.

du -hd1|sort -n
truncate -s 1k inputfile
xterm -e su -
ln -s ../lib-pcb packages
TERM=xterm-color
export TERM
oder:
TERM=cons25;    export TERM
export PACKAGEROOT="ftp://ftp2.de.freebsd.org"
remove lines with tab or space:
sed -e '/^[[:blank:]]*$/d' ./praktikum.txt &gt; prak3.txt
add " - " to the beginning ot each line
sed -e 's/.*/ - &amp;/' ./prak6 &gt; prak7
remove commas from a matlab data output:
sed -e 's/\,/ /g' 15.txt &gt; 16.txt
to add compile flags before ./configure:
export CFLAGS="$CFLAGS -I/usr/X11R6/include"

g11tch/shell_scripts_and_one_liners.md