Skip to content

Instantly share code, notes, and snippets.

@Cynnexis

Cynnexis/$ Bash Magic Spellbook.md

Last active March 18, 2025 02:29
Show Gist options
  • Save Cynnexis/9a3f1ba9ded48f6454fe5514df4c621a to your computer and use it in GitHub Desktop.
Save Cynnexis/9a3f1ba9ded48f6454fe5514df4c621a to your computer and use it in GitHub Desktop.
Download ZIP
✨ Bash Magic Spellbook
Raw
$ Bash Magic Spellbook.md

🧙✨ Secret Bash Magic Spellbook 🔮🪄🌟

This gist contains useful Bash spells that I acquired.

Table of Contents:

General

⛔ Script exits at first error

In Bash, you can put the following line at the top of your script, after the shebang:

set -euo pipefail

Explanation:

  • -e: If a command fail, then the script stops and fail too. This avoid the script to continue when critical commands are failing (like cd). The commands in until, while, for, if and elsif, but also the commands that are in a && or || expression are ignored, so conditions can still work.
  • -u: If shell encounters an undefined variable, the script will crash. Useful when the variables must be considered mandatory. Empty variable are not considered as undefined.
  • -o: Set a specific option on for this shell session. In this case, the option pipefail will cause the script to fail if a pipe command fails (like /bin/false | cat).

Source: Bash Reference Manual - The Set Builtin

🐛 Debugging a Bash script

Debugging a Bash script is hard without a good debugger, and echo commands can be a lot! So, in order to debug a Bash script quickly, please consider using the following spell that will output to the standard error every commands (after shell expansion) before execution:

PS4='+[${FUNCNAME[0]:-main}]${BASH_SOURCE[0]:-}:$LINENO> '
set -x

This will print all commands with the expanded prefix defined by PS4, that will indicate the current function from the stack (or default to main), the file that is being executed (useful if your script calls other sub-scripts) and the current line of the command. If you want to encapsulate your debugging code, you can close this debugging session with the following counter-spell:

{ set +x; } 2> /dev/null

The command set +x will remove the debugging property, but because we don't want it to be shown to the console (because it is superfluous), we encapsulate it under a sub-shell command (with {}) and redirect the standard error to null, so it is not displayed. The general command will not be outputed because when the shell would want to, set +x will already have stopped the debugging session.

In order to get better error message, you can also trap errors! See how to catch multiple traps to learn how to efficiently trap them!

Happy catching!

📨 Parse arguments

Parameters:

The following snippet shows how to parse arguments from the command line with long and short option, with or without the equal sign:

while [ $# -gt 0 ]; do
  case "$1" in
    --help|-h)
      print_help
      exit 0
      ;;
    --hostname=*)
      SRV_ADDR="${1#*=}"
      shift
      ;;
    --hostname)
      shift
      SRV_ADDR="$1"
      shift
      ;;
    --port=*)
      SRV_PORT="${1#*=}"
      shift
      ;;
    --port|-p)
      shift
      SRV_PORT="$1"
      shift
      ;;
    *)
      echo "Unknown argument: $1" 1>&2
      print_help 1>&2
      exit 1
      ;;
  esac
done

Parameters with files:

If you want the user to provide one or multiple files after specifying arguments, use the following snippet:

files=()
# Tell if the script must process arguments or files. It starts y processing
# arguments (true).
process_args='true'
while [ $# -gt 0 ]; do
  if [[ $process_args = 'false' ]]; then
    files+=("$1")
    shift
  else
    case "$1" in
      --help|-h)
        print_help
        exit 0
        ;;
      --hostname=*)
        SRV_ADDR="${1#*=}"
        shift
        ;;
      --hostname)
        shift
        SRV_ADDR="$1"
        shift
        ;;
      --port=*)
        SRV_PORT="${1#*=}"
        shift
        ;;
      --port|-p)
        shift
        SRV_PORT="$1"
        shift
        ;;
      --)
        shift
        process_args='false'
        ;;
      -*)
        echo "Unknown argument: $1" 1>&2
        print_help 1>&2
        exit 1
        ;;
      *)
        process_args='false'
        files+=("$1")
        shift
        ;;
    esac
  fi
done

📝 Pass arguments or read from stdin

In a bash script or function, sometimes you want to recieve an argument, or if nothing is passed, read it from stdin. The following spell will help you achieve this:

set -- "${1:-$(</dev/stdin)}" "${@:2}"

💂 Check if root

To check if a script is being executed as root, use the following spell:

if [ "$EUID" -ne 0 ]; then
  echo "This script needs to be run as root." 1>&2
  exit 1
fi

⏩ Manage script's redirection

If you want you script to automatically redirect all its own stdout and stderr output to a file, or a process, without changing the invocation of your script using a pipe, you can use the following spell:

exec 1> >(tee -a "$LOG_FILE") 2> >(tee -a "$LOG_FILE" 1>&2)

In this spell, both stdout and stderr will still be printed to the console, but also to an external file, so you can log your script's output.

Here's a more generic spell to write your script's output to a file and the console, without saving the colors:

# Function to use when redirecting stdout and stderr to a file and the console
_redirect() {
  tee >(sed -r "s/\x1B\[(([0-9]{1,2})?(;)?([0-9]{1,2})?)?[m,K,H,f,J]//g" | tee -a bug.log > /dev/null)
}

# Redirect all stdout and stderr to a file
exec 1> >(_redirect) 2> >(_redirect 1>&2)

To undo this action, you need to adapt it to save the default output and error first:

exec 3>&1 4>&2
exec 1> >(tee -a "$LOG_FILE") 2> >(tee -a "$LOG_FILE" 1>&2)

# ...

# Undo redirection
exec 1>&3 2>&4

⏳ Flush filesystem changes

⚠️ WARNING!

The following is considered dark magic and should be used with caution.

Sometimes, you will perform an operation on the filesystem, and then immediately after, you make an operation on those new changes, and the last command fails, reporting that the first changes you made are not there. This very strange bug happens in script, when two commands are executing sequentially very quickly.

To solve this, you can separate those two commands with sleep 0.

It basically tricks the scheduler to re-task your Bash script immediately after. But the real magic is that it also flushes the I/O queue.

Source: https://stackoverflow.com/a/7577647

🔍 Check array contains item

The following function will search

# Check if a bash array contains the given item.
#
# PARAMETERS
# ==========
# $1: The element to search in the array.
# $*: The array items. You can pass it as "${my_array[@]}" (with double quotes).
#
# RETURN CODES
# ============
# 0: The item is in the array.
# 1: The item was not found in the array.
array_contains() {
  local seeking=$1; shift
  local in=1
  for element; do
    if [[ $element == "$seeking" ]]; then
      in=0
      break
    fi
  done
  return $in
}

arr=(a b c "d e" f g)
array_contains "a b" "${arr[@]}" && echo yes || echo no    # no
array_contains "d e" "${arr[@]}" && echo yes || echo no    # yes

If you want to only pass the array name instead of its values:

# Check if a bash array contains the given item.
#
# PARAMETERS
# ==========
# $1: The name of the array. It will be expanded using the ${!parameter}
#     indirection.
# $2: The element to search in the array.
#
# RETURN CODES
# ============
# 0: The item is in the array.
# 1: The item was not found in the array.
array_contains() {
  local array="$1[@]"
  local seeking=$2
  local in=1
  for element in "${!array}"; do
    if [[ $element == "$seeking" ]]; then
      in=0
      break
    fi
  done
  return $in
}

# shellcheck disable=SC2034
arr=(a b c "d e" f g)
array_contains arr "a b" && echo yes || echo no    # no
array_contains arr "d e" && echo yes || echo no    # yes

Source: StackOverflow

🔠 Change text case

In Bash v4+, you can use the following spells:

To uppercase:

$ foo="bar BAR"
$ awk '{print toupper($0);}' <<< "$foo"
BAR BAR

Source: https://www.w3schools.io/terminal/bash-string-uppercase/

To lowercase:

$ foo="baR BAR"
$ awk '{print tolower($0);}' <<< "$foo"
bar bar

Source: https://www.w3schools.io/terminal/bash-string-uppercase/

Capitalize the first letter of a string:

$ foo="bar bar"
$ echo "${foo^}"
Bar bar

Source: https://stackoverflow.com/a/12487455

💣 Join array into a string

To join a bash array into a string, use the following spell:

cities=(Paris "New York" Madrid)
itinerary=$(IFS=, ; echo "${cities[*]}")
echo "My itinerary: $itinerary"

⚠ Note that IFS only accepts one character, it cannot support ", " for example.

Source: https://stackoverflow.com/a/9429887

💥 Split string into array

To split a string into a bash array, you can use this spell:

cities="Paris,Madrid,Rome"
IFS=',' read -ra array <<< "$cities"
echo "The first city to visit is ${array[0]}"

Source: https://stackoverflow.com/a/10586169

🥅 Multiple traps

The following variable and function create a stack for each used signal, so you can add multiple trap to the same signal:

# Associative arrays, the keys are the signal and the values the traps
declare -A _trap_stack

# Function to add a trap with a specific signal.
#
# PARAMETERS
# ==========
# $1: The trap to add. If '-' is given, the stack associated to the given signal
#     is reset by calling `trap - $2`.
# $2: The signal.
add_trap() {
  if [[ -z "${_trap_stack["$2"]:-}" ]]; then
    _trap_stack["$2"]="$1"
  else
    _trap_stack["$2"]="${_trap_stack["$2"]}; $1"
  fi

  # shellcheck disable=SC2064
  trap "${_trap_stack["$2"]}" "$2"
}

Example:

add_trap 'touch err1.txt' ERR
add_trap 'touch err2.txt' ERR

add_trap 'touch file1.txt' EXIT
add_trap 'touch file2.txt' EXIT

⚠ WARNING: Do NOT mix named and integer signals, they will overwrite each other.

To trap any eror and print as many information as possible about it:

# Trap errors to print as much information as possible
# shellcheck disable=SC2016
add_trap 'echo "Error: The error code $? was returned in ${BASH_SOURCE[0]} line ${LINENO} and in function \"${FUNCNAME[0]:-main}\" when executing \"${BASH_COMMAND}\"." 1>&2' ERR

🔁 Loop over a JSON list

Welcome to bash.js:

my_json='["a", "b", "c"]'
my_array=()

while IFS=$'\n' read -r item; do
  echo "Adding $item to my_array..."
  my_array+=("$item")
done < <(jq -Mrc '.[]' <<< "$my_json")

echo "My array:"
echo "${my_array[@]}"

To avoid any subprocess from consuming your shell input, one can use file descriptor:

exec 3< <(jq -Mrc '.[]' <<< "$my_json")
while IFS=$'\n' read -r item <&3; do
  my_command "$item"
done

# Close file descriptor
exec 3<&-

If you want to convert a JSON list to a Bash array in Bash, you can also use this trick:

readarray -d $'\n' -t my_array <<< "$(jq -Mrc '.[]' <<< "$my_json")"

➿ Parse files and folders with find

If you want to use a loop on find results, use the following spell:

while IFS= read -rd '' file
do
  echo "Playing file $file."
done < <(find mydir -mtime -7 -name '*.mp3' -print0)

➰ Iterate over character-separated string

If you have a comma-separated list of item in a variable, you can loop over it by using Bash substitution:

list=abc,def,ghi

for item in ${list//,/ }; do
  echo "$item"
done

Source: https://stackoverflow.com/a/35894538/7347145

⬇️ Get public IP address

To get the public IP address, use the following spell:

curl -fsSL https://ipinfo.io/ip

To save it in a variable:

my_ip=$(curl -fsSL https://ipinfo.io/ip 2> /dev/null | tr -d '\n')

Source: Linux Config

🔏 Encrypt/Decrypt a file/folder

To encrypt and decrypt a file using GPG:

gpg --output encrypted.data --symmetric --cipher-algo AES256 un_encrypted.data
gpg --output un_encrypted.data --decrypt encrypted.data

To encrypt and decrypt multiple files and/or folder(s), zip them in a .tar file before:

# Zip
tar czf myfiles.tar.gz file1 file2 mydirectory/
# Encrypt
gpg --output myfiles.tar.gz.enc --symmetric --cipher-algo AES256 myfiles.tar.gz

# Decrypt
gpg --output myfiles.tar.gz --decrypt myfiles.tar.gz.enc
# Unzip
tar xzf myfiles.tar.gz

For more information, see this gist.

👇 Source a bash file in the same location of the script

source "$(dirname "$0")/my_deps.bash"

🔒 Locking file

This snippet allows your script to have a unique run, and avoid two execution of the same script, with a fancy error message.

# Locking file
LOCK_FILE_PATH="/var/tmp/$(basename $0).lock"
if [[ -f $LOCK_FILE_PATH ]]; then
  set +eu
  user=$(yq -r '.user' < "$LOCK_FILE_PATH")
  uid=$(yq -r '.uid' < "$LOCK_FILE_PATH")
  info=$(yq -r '.info' < "$LOCK_FILE_PATH")
  date_iso8601=$(yq -r '.date.iso8601' < "$LOCK_FILE_PATH")
  date_now=$(date +%s)
  date_then=$(yq -r '.date.unix_epoch_s' < "$LOCK_FILE_PATH")
  timedelta=$(( date_now - date_then ))
  echo -e "ERROR: It looks like the script is already being run by $user($uid) $info.\nThis run started at $date_iso8601 ($(date "-d@$timedelta" -u +%H:%M:%S) from now).\nIf you think this is a mistake, you can remove the lock file at \"$LOCK_FILE_PATH\"." 1>&2
  exit 1
fi

# Trap EXIT to remove the lock file
trap 'rm -f "$LOCK_FILE_PATH"' EXIT

# Write the lock file (YAML syntax)
cat <<EOF > "$LOCK_FILE_PATH"
---
# Lock file for $0.
user: "$USER"
uid: $UID
info: "$USER_INFO"
date:
  unix_epoch_s: $(date +%s)
  iso8601: "$(date '+%FT%T')"
EOF

💽 See disk usage

This section will teach you useful spells to analyze the space of your computer and invidual folders with their sub-directories.

Disk

To list the avaialble space on each file systems, use the df(1) spell:

df -h

Folders

To analyze the spaces taken by the files by summing them by directories, use the du(1) spell:

du -csh *

This will comptue the totla size of all folders in the current direcotyr, and display a summary of the total space.

Big files

You can search for big files using the following commands:

sudo find /bin /sbin /usr /etc /home /opt /root /var/log -type f -size +10M

To search for duplciated filed using fdupes(1):

sudo fdupes -r /bin /sbin /usr /etc /home /opt /root /var/log

Optimize disk

Remove useless packages from APT:

sudo apt-get autoremove

Free some logs:

sudo journalctl --disk-usage
sudo journalctl --vacuum-time=3d

Source: https://itsfoss.com/free-up-space-ubuntu-linux/

You can also clean up the temporary files using the following spell:

find /tmp -mtime +7 -and -not -exec fuser -s {} \; -and -exec rm -rf {} \;

Source: Super User

⬇ Import all aliases in non-interactive shell

In non-interactive environment, such as specific bash scripts, you cannot use aliases, which is a shame when you want to cast some overly-complicated spells. The following snippet allows you to load them back in a bash script with some shopt magic and parsing:

shopt -s expand_aliases
# ... or pass "-O expand_aliases" to the bash invocation

while read -r line; do
  eval \$line
done < <(grep -Pe '^\s*alias\s+' ~/.bashrc)

eval my_alias

🔗 Extract hostname from simple URL

url=https://my.example.com/
hostname="${url/#http:\/\//}"
hostname="${hostname/#https:\/\//}"
hostname="${hostname/%\//}"

▶ Add prefix to all lines from stdout and stderr of a command

If you want to prefix all the outputed lines of a command to the console, use the following pipe:

LOG_FILE="$(date +%F-%Hh%Mm%Ss)_my_command.log"
my_command |& tee -a "$LOG_FILE" | stdbuf -o0 sed 's@^@my_prefix> @'; echo "my_command exited with status ${PIPESTATUS[0]}." | tee -a "$LOG_FILE"; }

📃 Save lines from stdout to an array

IFS=$'\n' lines=($(head -n10 my_file.txt))

echo "${#lines[@]}" # 10

📩 Add lines to a file after a regex

sed '/my_regex/r add.txt' file.txt

with add.txt:

new line 1
new line 2
new line 3

and file.txt:

My file
my_regex
Final line

Results:

My file
my_regex
new line 1
new line 2
new line 3
Final line

Source: https://stackoverflow.com/a/22497499

⏩ Command completion from history

In Bash, to re-execute a command you already launched with the beginning of it, you can type Ctrl+R to search through the history. One way to do it easier like in Zsh is to add the following content to your ~/.inputrc:

# Key bindings, up/down arrow searches through history
"\e[A": history-search-backward
"\e[B": history-search-forward
"\eOA": history-search-backward
"\eOB": history-search-forward

You can then load the configuration with: bind -f ~/.inputrc

Source: https://unix.stackexchange.com/a/20830

🌳 Print folders hierarchy

To print the hierarchy of a folder, you can use the following spell:

tree

If tree is not installed, use this:

find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'

Docker

🐳 Execute a command from the host in the container

If you have a binary on your host that you would like to run in the container, you can use the following spell:

  1. Get the PID of the container: 
    docker inspect --format '{{.State.Pid}}' <container>
  2. Execute your command using nsenter(1): 
    nsenter -t <pid> -n <command>

🐳 Extract image name from TAR

The complete name (<repo>:<tag>) of a Docker image is saved in the TAR file when exported. To fetch it, use the following spells:

# Get the list of repositories
repos=$(tar -xf "$main_file_path" -O repositories)

# Extract the first repo name
repo=$(jq -Mr 'keys[0]' <<< "$repos")

# Get the tag from the repo object
tag=$(jq -Mr \
  --arg repo "$repo" \
  '.[$repo] | keys[0]'
  <<< "$repos")

# Get the complete Docker image name
name="$repo:$tag"

🐳 Dependency-based Docker tag hash

If you want to have a script that automatically build your Dockerfile if the image is missing or some critical files have changed, but without re-building it every time you use it, here the solution:

You can hash the content of those critical files and build arguments and put them in the tag part of the image name.

tag="$(echo "author=$UID" \
  | cat - "Dockerfile" "entry-point.sh" \
  | sha256sum \
  | awk '{print $1;}')"

docker build -t "my_image:$tag" --build-arg "author=$UID" .

Git

◀ Revert a commit without creating a commit

Sometimes, we'd like to revert some changes from a previous commit without generate a reverse-commit. One can use git revert --no-commit, but it will stage the file. In order to revert a commit changes without staging them, here a pipe spell you can cast:

git show <rev> | git apply -R

Source: StackOverflow

🌿 Create a disconnected branch from default code

If you want to create a branch that is completely empty, without any link with previous commits or refs, you can use the following spells:

git checkout --orphan my-branch
git rm -rf .
# <add files>
git add $files
git commit -m 'Initial commit for my-branch'

This is useful when you want to seperate your code from your documentation for example.

Source: https://stackoverflow.com/a/5690048/7347145

SSH

⌚ Synchronize date with SSH

Send date to server:

ssh -t $remote_server sudo date -s "@$(date -u +"%s")"

Source: https://www.commandlinefu.com/commands/view/14135/synchronize-date-and-time-with-a-server-over-ssh

Get date from server:

sudo date "--set=$(ssh $remote_server date)"

Source: https://unix.stackexchange.com/a/218917/436587

🛑 Escape a SSH session

If you want to quickly evade an SSH session or it's frozen, and you can get away, you can use the following key shortcut to kill it: Enter+~+.

Source: https://stackoverflow.com/a/28981113/7347145

🐱‍👤 Execute SSH Agent at startup

To execute the SSH agent when you login, you can add the following snippet to your ~/.bashrc, ~/.bash_profile or to a system profile script like /etc/profile.d/ssh-agent.sh:

# Launch ssh-agent at startup.
# Source code inspired from:
# * https://unix.stackexchange.com/a/132117
# * https://code.visualstudio.com/docs/remote/troubleshooting#_setting-up-the-ssh-agent

if [ -z "$SSH_AUTH_SOCK" ]; then
  # Check for a currently running instance of the agent
  running_agent="$(ps -ax | grep 'ssh-agent -s' | grep -v grep | wc -l | tr -d '[:space:]')"
  if [ "$running_agent" = "0" || ! -f /tmp/ssh-agent.sh ]; then
    ssh-agent -s < /dev/null 2> /dev/null > /tmp/ssh-agent.sh
    chmod +x /tmp/ssh-agent.sh
  fi
  source /tmp/ssh-agent.sh > /dev/null
fi

Source:

🧦 Create SSH socket

You can create your own socket on your local filesystem to allow applications to communicate with remote server through SSH. This can be done by casting the following spell:

ssh -fnNTL /path/to/local/socket:localhost:22 [email protected]

Explanation:

  • -f: Go to background just before command execution.
  • -n: Prevents reading from stdin.
  • -N: Do not execute a remote command, just forward the connection.
  • -T: Disable pseudo-tty allocation.
  • -L: Forward local port to remote socket, with the format [bind_address:]port:host:hostport.

GitHub Actions

🌈 Colorize commands

To colorize the output of your commands in GitHub Actions, you can use the following spell:

# shellcheck disable=SC2016
color_cmd='echo -e "\033[0;90m\$ \033[0;36m${BASH_COMMAND}\033[0m" 1>&2'
# shellcheck disable=SC2064
trap "$color_cmd" DEBUG
curl wttr.in
{ trap - DEBUG; } 2> /dev/null

echo "Done"

See all color codes here.

🥅 Trap error

This spell will trap error in a bash script with the GitHub Actions error command:

# Trap errors to print as much information as possible
trap 'echo "::error file=${BASH_SOURCE[0]:-},line=${LINENO:-},title=Error-$?:: The error code $? was returned when executing \"${BASH_COMMAND}\"." 1>&2' ERR

WSL

📋 Use Windows Clipboard

You can use the Windows clipboard system in the WSL to perform copy-paste operations. The following instructions will show you how to cast some ctrl+c/ctrl+v spells inside a WSL terminal:

Copy:

cat my_clipboard.txt | clip.exe
my_command |& clip.exe

Paste:

powershell.exe -c Get-Clipboard > my_clipboard.txt
powershell.exe -c Get-Clipboard | my_command

Source: SuperUser

🔊 Play beep from WSL

The WSL cannot play any sounds because it is not connected to any audio interface. However, you can trick it to play a sound by calling powershell.exe and use the built-in beep function. In a WSL console, cast the following spell:

powershell.exe "[console]::beep(500,300)"

The first argument defines the pitch (must be between 190 and 8500 to be heard), and the second argument is the duration in milliseconds.

You can also set it as a function to be more confortable:

pbeep () {
  timeout 5s powershell.exe "[console]::beep(${1:-500},${2:-300})"
}

Source: Microsoft Dev Blogs

🔁 Convert UNIX path to Windows

If you want to convert a UNIX path to a Windows path, you can use the following spell:

sed -re 's|^/(mnt/)?([c-z])/|\U\2:\\|' -e 's|^/|C:\\|' -e 's|/|\\|g'

It handles the /mnt/c/ and /c/ prefixes, and converts the slashes to backslashes.

⏳ Start cron daemon at Windows startup

In WSL, configure sudo by executing sudo visudo and adding the following line:

# Allow anyone to start the cron daemon
%sudo ALL=NOPASSWD: /etc/init.d/cron start

In Windows, open the applications menu with Super and search for WSL. Right-click on the icon to select "Open file location". In the explorer, right-click on the wsl.exe file and select "Copy". Close the explorer.

Then, use the shortcut Super+R and type shell:startup in the dialog box. It will open the folder containing all softwares and script to execute at startup. Right-click and select "Past shortcut". Open the properties of the shortcut, and add the following argument in the target, after the location of the executable file: sudo /etc/init.d/cron start

The target should look something like:

"C:\Program Files\WindowsApps\wsl.exe" sudo /etc/init.d/cron start

Source: https://blog.snowme34.com/post/schedule-tasks-using-crontab-on-windows-10-with-wsl/index.html

🐳 Install Docker engine in WSL

First, uninstall Docker Desktop for Windows by going to Settings > Apps, type "Docker" in the search bar and then click on "Uninstall".

Install WSL

Then, make sure you have WSL installed (you can use the MS-DOS command wsl -l -v). If not, follow those steps:

  1. Download Ubuntu from the Microsoft Store.
  2. Open a terminal by typing "Cmd" in the Windows search bar, and enter the following command: wsl -l -v You should see something like this (make sure the version is 2): 
      NAME      STATE           VERSION

* Ubuntu    Running         2
    If you see multiple images, you can set a default one by typing wsl set default <Name> (with <Name> being the name (first column) of the image, like "Ubuntu").
  3. You can now set up a user in the WSL you just installed by typing wsl or opening the WSL terminal from the Windows search bar.
  4. If you encountered a problem at some point, please read the official documentation.

Install Docker in WSL

Now, you can install Docker inside the WSL:

  1. Open a WSL terminal (through the Windows CMD or the WSL terminal)
  2. Download the installation script and execute it: 
    curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
    Ignore any warning concerning Docker and WSL.
  3. Add your default user to the docker group, so all subsequent docker commands can be executed without root privileges: 
    sudo usermod -aG docker $USER
  4. Make sure the Docker Compose plugin is installed, in Debian-based distribution you can type: 
    sudo apt-get update
sudo apt-get install docker-compose-plugin
  5. Finally, make sure that your Linux distro uses the legacy iptables for Docker network isolation: 
    sudo update-alternatives --config iptables

Configure Docker

Now that Docker is installed, it is time to configure it.

  1. First, make sure to remove the directory .docker in your home folder in the WSL: rm -rf ~/.docker
  2. Then, create it back again, and add a default configuration file: 
    mkdir ~/.docker
echo "{}" > ~/.docker/config.json
  3. Now, you can configure the Docker daemon. Make sure the Docker directory exists (sudo mkdir -p /etc/docker) and then edit the file /etc/docker/daemon.json to enter the following configuration: 
    {
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "features": {
    "buildkit": true
  },
  "experimental": false,
  "hosts": [
    "unix:///var/run/docker.sock"
  ]
}
    You can personalize it if you want, but make sure that the hosts property exists (because systemctl won't configure it for you) and is configured accordingly to your Docker client, read the official documentation for more details.
  4. Now that Docker is installed, you need to start the Docker Engine. You do that manually, or make sure it launches at Windows startup to avoid doing it manually. To start it up manually, enter the following command: 
    sudo nohup /usr/bin/dockerd < /dev/null &

Finally, make sure that Docker is properly configured and running by typing the following command:

docker version

🐳 Start dockerd daemon at Windows startup

To install Docker on the WSL without Docker Desktop: https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop

In WSL, configure sudo by executing sudo visudo and adding the following line:

# Allow anyone to start the docker daemon
%sudo ALL=NOPASSWD: /root/start-docker.sh

Then, add the file /root/start-docker.sh with the following content:

#!/bin/bash
nohup /usr/bin/dockerd < /dev/null &> /var/log/dockerd.log &

Note that the stdout and stderr redirection is optional but recommended, otherwise nohup will create the nohup.out file in /root/.

Don't forget to set the execution permission with sudo chmod 0755 /root/start-docker.sh.

Make sure to create the log file with the correct permissions:

sudo mkdir -p /var/log
sudo touch /var/log/dockerd.log
sudo chown root:docker /var/log/dockerd.log
sudo chmod 0664 /var/log/dockerd.log

Because dockerd will be launched from a custom script and not systemd, we need to configure the Docker daemon by editing the file /etc/docker/daemon.json with the following content:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "features": {
    "buildkit": true
  },
  "experimental": false,
  "hosts": [
    "unix:///var/run/docker.sock"
  ]
}

In Windows, open the applications menu with Super and search for WSL. Right-click on the icon to select "Open file location". In the explorer, right-click on the wsl.exe file and select "Copy". Close the explorer.

Then, use the shortcut Super+R and type shell:startup in the dialog box. It will open the folder containing all softwares and script to execute at startup. Right-click and select "Past shortcut". Open the properties of the shortcut, and add the following argument in the target, after the location of the executable file: sudo /root/start-docker.sh

The target should look something like:

"C:\Program Files\WindowsApps\wsl.exe" sudo /root/start-docker.sh
Raw
logging.bash
#!/bin/bash
# Check if terminal is TTY.
# Source code inspired from https://unix.stackexchange.com/a/10065/436587 by Mikel.
#
# EXIT CODES
# ==========
# 0: The terminal is TTY.
# 1: The terminal is not TTY.
is_tty() {
set +e
ncolors=$(TERM="${TERM:-dumb}" tput colors)
test -n "$ncolors" && test "$ncolors" -ge 8
ec=$?
set -e
return $ec
}
# Echo the first argument only if the terminal is TTY. If not and a second
# argument is given, the second argument is printed.
#
# PARAMETERS
# ==========
# $1: The string to output only in TTY. If not in TTY, the string won't be
# printed.
# $2: (Optional) The string to output if NOT in TTY. If the terminal is TTY,
# the argument $1 is printed, and $2 is ignored. If not in TTY, $2 will
# be printed if it is passed, or nothing is printed if only $1 is passed,
#
# OUTPUT
# ======
# stdout: The argument $1 or $2 depending on wether the terminal is TTY and if
# $2 has been passe dor not.
# stderr: Argument error (see exit codes).
#
# EXIT CODES
# ==========
# 0: Success (regardless of TTY or not, and if one or two arguments have been
# passed).
# 1: Argument error, zero arguments or more than two arguments have been given.
echo_tty() {
if [[ $# == 0 || $# -gt 2 ]]; then
echo "Error: Excepted 1 or 2 arguments, got $#." 2>&1
return 1
fi
if is_tty; then
echo -e "$1"
elif [[ $# == 2 ]]; then
echo -e "$2"
fi
}
# Log the given string according in the given color if TTY.
#
# PARAMETERS
# ==========
#
# $1: The color code, like "32" for "\e[32m" (green) or "0;32" for "\e[0;32m"
# (reset, then green).
# $*: The string to log. If not given, it is read from stdin.
_log_color() {
pre=
suf=
# Check if TTY
if is_tty; then
pre="\e[$1m"
suf="\e[0m"
fi
shift
set -- "${1:-$(</dev/stdin)}" "${@:2}"
echo -e "${pre}${*}${suf}"
}
# Log to the console really unimportant information.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_finer() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "1;30" "$*"
}
# Log to the console unimportant information.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_fine() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "1;35" "$*"
}
# Log to the console an information.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_info() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "0;34" "$*"
}
# Log to the console a valid message.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_valid() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "0;32" "$*"
}
# Log to the console a warning.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_warning() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "0;33" "$*" 1>&2
}
# Log to the console an error to standard error.
#
# PARAMETERS
# ==========
#
# $*: The string to log. If not given, it is read from stdin.
log_error() {
set -- "${1:-$(</dev/stdin)}" "${@:2}"
_log_color "0;31" "$*" 1>&2
}
# Log to the console a box with the given text inside.
#
# PARAMETERS
# ==========
# $1: The color of the box and text.
# $2: The right-top corner character.
# $3: The left-top corner character.
# $4: The left-bottom corner character.
# $5: The right-bottom corner character.
# $6: The horizontal bar character.
# $7: The vertical bar character.
# $*: The string to log.
#
# OUTPUT
# ======
# stdout: Print the box with its text.
log_box() {
echo
color=$1
shift
rt=$1
shift
lt=$1
shift
lb=$1
shift
rb=$1
shift
hz=$1
shift
vt=$1
shift
# Construct horizontal lines
horizontal_lines=''
horizontal_lines_length=0
# Add two more for text whitespaces
while [[ $horizontal_lines_length -lt $(( $(wc --chars <<< "$*") + 1)) ]]; do
horizontal_lines="$horizontal_lines$hz"
horizontal_lines_length=$(( horizontal_lines_length + 1 ))
done
_log_color "$color" "$rt$horizontal_lines$lt"
_log_color "$color" "$(printf "$vt %s $vt" "$*")"
_log_color "$color" "$rb$horizontal_lines$lb"
echo
}
log_expanded_box() {
term_width=$(/usr/bin/tput cols)
text_width=$(wc --chars <<< "$8")
text_lr_space=$(( ((term_width - text_width) / 2) - 2 ))
log_box "${@:1:7}" "$(printf "%*s%s%*s" "$text_lr_space" ' ' "$8" "$text_lr_space" ' ')"
}
# Log to the console a simple box with the given text inside.
#
# PARAMETERS
# ==========
# $1: The color of the box and text.
# $*: The string to log. If not given, it is read from stdin.
#
# OUTPUT
# ======
# stdout: Print the simple box with its text.
log_simple_box() {
log_box "$1" '┌' '┐' '┘' '└' '─' '│' "${*:2}"
}
# Log to the console a simple box with the given text inside, center-aligned.
#
# PARAMETERS
# ==========
# $1: The color of the box and text.
# $*: The string to log. If not given, it is read from stdin.
#
# OUTPUT
# ======
# stdout: Print the simple box with its text.
log_expanded_simple_box() {
log_expanded_box "$1" '┌' '┐' '┘' '└' '─' '│' "${*:2}"
}
# Log to the console a double box with the given text inside.
#
# PARAMETERS
# ==========
# $1: The color of the box and text.
# $*: The string to log. If not given, it is read from stdin.
#
# OUTPUT
# ======
# stdout: Print the double box with its text.
log_double_box() {
log_box "$1" '╔' '╗' '╝' '╚' '═' '║' "${*:2}"
}
# Log to the console a double box with the given text inside, center-aligned.
#
# PARAMETERS
# ==========
# $1: The color of the box and text.
# $*: The string to log. If not given, it is read from stdin.
#
# OUTPUT
# ======
# stdout: Print the double box with its text.
log_expanded_double_box() {
log_expanded_box "$1" '╔' '╗' '╝' '╚' '═' '║' "${*:2}"
}
# Time the given command and parameter with the best found tool on the system.
#
# PARAMETERS
# ==========
# $0: Command to execute
# ${*:2}: Additionnal parameters to pass to the given command.
#
# ENVIRONMENT VARIABLE
# ====================
# time_fmt: The format of the time elapsed in standard output. Defaults to
# 'Time elapsed: ' (and then depends on the found tool).
#
# OUTPUT
# ======
# stdout: Same output as the command, followed by the time.
# stderr: Same output as the command.
#
# EXIT CODES
# ==========
# Same as command
ms_time() {
if [[ -x /usr/bin/time ]]; then # Check if time tool is installed
time_fmt="${time_fmt:-"Time elapsed: %E"}"
/usr/bin/time -f "$time_fmt" "$@"
return $?
elif command -v time | grep -qPe '^time$'; then # built-in time tool
time_fmt="${time_fmt:-"Time elapsed: %R"}"
TIMEFORMAT="$time_fmt" time "$@"
return $?
else # manual
start=$(date +%s)
"$@"
ec=$?
end=$(date +%s)
time_elapsed="$(( end - start ))"
time_elapsed_fmt=$(date "-d@$time_elapsed" -u +%H:%M:%S)
echo "Time elapsed: $time_elapsed_fmt"
return $ec
fi
}
export -f is_tty
export -f echo_tty
export -f _log_color
export -f log_finer
export -f log_fine
export -f log_info
export -f log_valid
export -f log_warning
export -f log_error
export -f log_box
export -f log_expanded_box
export -f log_simple_box
export -f log_expanded_simple_box
export -f log_double_box
export -f log_expanded_double_box
export -f ms_time
Raw
stripcolors.sh
#!/bin/bash
set -euo pipefail
sed -r "s/\x1B\[(([0-9]{1,2})?(;)?([0-9]{1,2})?)?[m,K,H,f,J]//g" "$@"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment