GNU Parallel Mastery: The Ultimate Cheat Sheet

parallel.md

🚀 GNU Parallel Mastery: The Ultimate Cheat Sheet

📊 Basic Usage

Command	Description	Example
parallel echo ::: A B C	🔤 Process items in parallel	Output: A, B, C (in any order)
parallel echo {} ::: *.txt	🔍 Use {} as placeholder	Echoes names of all .txt files
cat file.txt | parallel echo	📥 Read input from stdin	Processes each line of file.txt
parallel -j4 command ::: item1 item2 item3	🔢 Limit to 4 jobs at a time	Runs 'command' on items, max 4 parallel

🔧 Input Sources

Syntax	Description	Example
:::	📊 Command line input	parallel echo ::: A B C
::::	📂 File input (one item per line)	parallel echo :::: items.txt
:::+	🔗 Combine arguments from multiple sources	parallel echo ::: A B C :::+ X Y Z
-a file	📄 Read input from file	parallel -a items.txt echo

🔀 Job Control

Option	Description	Example
-j N	🔢 Run N jobs in parallel	parallel -j4 command ::: 1 2 3 4 5 6
-k	🔤 Keep output in order	parallel -k echo ::: C B A
--halt soon,fail=1	🛑 Stop all jobs on first failure	parallel --halt soon,fail=1 command ::: 1 2 3
--eta	⏱️ Show estimated time of completion	parallel --eta command ::: {1..100}
--bar	📊 Display progress bar	parallel --bar command ::: {1..100}

🔁 Replacement Strings

String	Description	Example
{}	🔄 Input line	parallel echo {} ::: A B C
{.}	🔪 Input line without extension	parallel echo {.} ::: file.txt
{/}	📂 Basename of input line	parallel echo {/} ::: /path/to/file.txt
{//}	📁 Dirname of input line	parallel echo {//} ::: /path/to/file.txt
{#}	🔢 Sequence number of job	parallel echo {#} {} ::: A B C
{%}	🔢 Job slot number	parallel -j2 echo {%} {} ::: A B C D

📡 Remote Execution

Command	Description	Example
--sshlogin user@host1,user@host2	🖥️ Run on remote hosts	parallel --sshlogin user@host1,user@host2 command ::: 1 2 3
--transfer	📤 Transfer files to remote	parallel --transfer --sshlogin user@host command ::: file1 file2
--return {.}.out	📥 Return output files from remote	parallel --return {.}.out --sshlogin user@host command ::: file1 file2

🔧 Advanced Techniques

Technique	Description	Example
--pipe	🚿 Split input stream	cat bigfile.txt | parallel --pipe grep 'pattern'
--xargs	🔨 Emulate xargs	find . -name '*.jpg' | parallel --xargs convert {} {.}.png
--shebang	📜 Use in scripts	#!/usr/bin/parallel --shebang
--dry-run	🏃‍♂️ Show commands without executing	parallel --dry-run echo ::: A B C
--results outdir	📁 Store output in directory	parallel --results outdir command ::: 1 2 3

🏆 Pro Tips

1. 📚 Use man parallel_tutorial for an extensive guide
2. 🔍 Use parallel --citation in your publications if you use GNU Parallel
3. 🔢 Use {1}, {2}, etc. for multiple input sources
4. 📊 Use --joblog logfile to keep a log of executed jobs
5. 🔄 Use --resume to resume interrupted runs
6. 🔒 Use --compress when transferring large amounts of data to remote hosts
7. 🔧 Use --tag to prefix each line of output with the input line
8. 📈 Use --progress for a live progress update
9. 🔍 Use --regexp to use regular expressions in replacement strings
10. 🔀 Use parallel --link to process input sources in lockstep

📜 Example: Image Processing

#!/usr/bin/env bash

# Resize all .jpg images to 50% of their original size
find . -name "*.jpg" | parallel convert {} -resize 50% {.}_resized.jpg

# Convert all .png images to .jpg in parallel, utilizing all CPU cores
parallel convert {} {.}.jpg ::: *.png

# Process images on remote servers
parallel --sshlogin user@server1,user@server2 --transfer \
    "convert {} -resize 50% {.}_resized.jpg && rm {}" ::: *.jpg

# Generate thumbnails for all images, showing progress
parallel --bar "convert {} -resize 100x100^ -gravity center -extent 100x100 {}_thumb.jpg" ::: *.jpg

# Process large log file in parallel
cat large_log.txt | parallel --pipe grep 'ERROR' > errors.txt

# Parallel download of files
cat urls.txt | parallel --eta wget {}

Remember, GNU Parallel is a powerful tool. Use it responsibly and always test your commands with --dry-run before executing them on important data! 🚀💻