bash toolkit

bash.md

Map

replaces lines with a substring match of that line (capture groups)

sed -E 's/regex/string/'

echo "a secret: 12345" | sed -E 's/a secret: (.*)/\1/'

(outputs 12345)

---

Filter

filtering by line content (use for unstructured text)

• grep -E <regexp> file
• grep -Ev <regexp> file

filter by range of line numbers

• aws 'FNR >= <LINE START> && FNR <= <LINE END> { print }'
• ex: awk 'FNR >= 3 && FNR < 5 {print}' foo.txt
• ex: awk 'FNR > 10 {print}' foo.txt

SQL-like filtering (for more complex things, when underlying data is structured)

awk '<conditions> { print <columns> }'

(analogous to SELECT <columns from x WHERE <conditions>)

• awk '$2 ~ /^foo/ { print $1 }'
• awk '$1 !~ /baz$/ { print $1 }'
• awk '$2 == "a" || $3 == 42 { print $1 }'
• awk '{ print $2" "$4; }'

Examples

cat <<EOT | awk 'NR > 1 { print }'
a b c d e
1 2 3 4 5
1 2 3 3 3
1 2 4 5 1
EOT

# 1 2 3 4 5
# 1 2 3 3 3
# 1 2 4 5 1

cat <<EOT | awk 'NR > 1 && $3 == 3 { print }'
a b c d e
1 2 3 4 5
1 2 3 3 3
1 2 4 5 1
EOT

# 1 2 3 4 5
# 1 2 3 3 3

cat <<EOT | awk 'NR > 1 && $5 ~ /^orang/ { print }'
a b c d e
1 2 3 4 apples
1 2 3 3 oranges
1 2 4 5 orangutans
1 2 4 5 kittens
EOT

# 1 2 3 3 oranges
# 1 2 4 5 orangutans

cat <<EOT | awk -F ',' 'NR > 1 && $5 ~ /^orang/ { print }'
a,b,c,d,e
1,2,3,4,apples
1,2,3,3,oranges
1,2,4,5,orangutans
1,2,4,5,kittens
EOT

# 1,2,3,3,oranges
# 1,2,4,5,orangutans

---

Reduce

join two files together (concat)

cat f1 f2

cat <$(... bash command ...) <$(... bash command ...)

join two files together line-by-line (final result has columns from f1, f2)

paste f1 f2

join lines together using delimiter (final result is a single line)

printf "a\nb\nc\n" | paste -s -d ',' -

General Tips

use bash -E

use sed -E

quote the arguments!