URL Encoding in pure bash / sed.

._README.md

Credit to @jkishner for https://gist.github.com/jkishner/2fccb24640a27c2d7ac9

Also interesting: https://gist.github.com/cdown/1163649

url_encode.sh

#!/usr/bin/env bash

#
# Credit to @jkishner for https://gist.github.com/jkishner/2fccb24640a27c2d7ac9
#
# Also interesting: https://gist.github.com/cdown/1163649
#

function url_encode() {
    echo "$@" \
    | sed \
        -e 's/%/%25/g' \
        -e 's/ /%20/g' \
        -e 's/!/%21/g' \
        -e 's/"/%22/g' \
        -e "s/'/%27/g" \
        -e 's/#/%23/g' \
        -e 's/(/%28/g' \
        -e 's/)/%29/g' \
        -e 's/+/%2b/g' \
        -e 's/,/%2c/g' \
        -e 's/-/%2d/g' \
        -e 's/:/%3a/g' \
        -e 's/;/%3b/g' \
        -e 's/?/%3f/g' \
        -e 's/@/%40/g' \
        -e 's/\$/%24/g' \
        -e 's/\&/%26/g' \
        -e 's/\*/%2a/g' \
        -e 's/\./%2e/g' \
        -e 's/\//%2f/g' \
        -e 's/\[/%5b/g' \
        -e 's/\\/%5c/g' \
        -e 's/\]/%5d/g' \
        -e 's/\^/%5e/g' \
        -e 's/_/%5f/g' \
        -e 's/`/%60/g' \
        -e 's/{/%7b/g' \
        -e 's/|/%7c/g' \
        -e 's/}/%7d/g' \
        -e 's/~/%7e/g'
}

# Only invoke url_encode if the script is being executed
# (rather than sourced).
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    url_encode $@
fi

I just figured out a way to use curl to do the encoding:

input="Some string/to+url?encode@"
output=$(curl -s -w '%{url_effective}\n' -G / --data-urlencode "=$input" | cut -c 3-)
echo "$output"

Some+string%2Fto%2Burl%3Fencode%40

Of course, if you were encoding the data in order to use it with curl, then you can use --data-urlencode and skip this step. :-)

Tested on curl 7.68.0 and 7.79.1.

An earlier version of this comment used url instead of url_effective and that failed on 7.68.0 but worked on 7.79.1. Sorry if that affected you!

Nice trick to encode url with curl. Thanks @PenelopeFudd!

Running inside of an Alpine container I ran into problems with the curl method and the sed method seemed incomplete to me, so since we were running a Python project, I ended up with this:

$ python3 -c "from urllib.parse import quote; print(quote('${MY_SPECIAL_STRING}'))"

This'll barf if you have a ' in that string, but outside of that, it should work.

Maybe pipe it in via stdin?

Running inside of an Alpine container I ran into problems with the curl method and the sed method seemed incomplete to me, so since we were running a Python project, I ended up with this:

$ python3 -c "from urllib.parse import quote; print(quote('${MY_SPECIAL_STRING}'))"

This'll barf if you have a ' in that string, but outside of that, it should work.

You could pass in the value as a command-line argument to skip the quoting gymnastics:

$ MY_SPECIAL_STRING="\$foo'bar?" ; echo "${MY_SPECIAL_STRING}" ;  python3 -c 'import sys, urllib.parse ; print(urllib.parse.quote(sys.argv[1]))' "${MY_SPECIAL_STRING}"
$foo'bar?
%24foo%27bar%3F

if you have jq

echo "a b" | jq "@uri" -jRr

Oh, yes, jq is even better than curl at this, mainly because it's shorter, only uses one -, and the curl trick requires a fairly recent version.

Plus, it has a jRr token, which makes me think of hobbits! :-)