Using curl

curl_en.md

Using curl

Installation

• Linux (Ubuntu):

  apt install curl
  

• macOS: curl is bundled with the operating system, thus, there is no need to install it.

• Windows: https://curl.se/windows/

  • curl is bundled with Windows 10 and Windows 11, however, the build shipped with the operating system lacks some features and capabilities (e.g., support for HTTP/2 and HTTP/3), see here.

  • It should also be noted that curl is also bundled with Git for Windows, thus, it is available in Git BASH. However, it also lacks some features, such as support for HTTP/2 and HTTP/3.

  • In PowerShell, the curl command is an alias for a cmdlet that has nothing to do with the curl tool. Thus, you can refer to the curl tool as curl.exe in PowerShell.

• Installing with Miniconda (Linux, macOS, Windows):

  conda install curl
  

Trying Out the Examples

The commands in this document are intended to be executed in Bash, however, most of them also work on the Windows command line without any changes. The remaining commands can be easily adapted to the Windows command line by using the ^ symbol for escaping special characters instead of a \.

Checking Availability

curl
curl --version
curl -V

Help

Display available command line options:

curl --help
curl -h

Display manual:

curl --manual
curl -M
man curl

Documentation: https://curl.se/docs/

Basic Use

Website: http://ip-api.com/

curl ip-api.com/json
curl http://ip-api.com/json
curl http://ip-api.com/json -O
curl http://ip-api.com/json -O -s
curl http://ip-api.com/json -o ip-api.json -s
curl http://ip-api.com/json -o ip-api.json -v -s
curl http://ip-api.com/json -o ip-api.json --dump-header ip-api.txt -s
curl http://ip-api.com/json -o ip-api.json --trace - | less
curl http://ip-api.com/json -o ip-api.json --trace-ascii - | less

Redirection

Example (1)

Website: http://www.w3.org/

curl http://w3.org -v
curl http://w3.org -v --location -o w3.html
curl http://w3.org -v -L -o w3.html

Example (2)

URL shortening service: https://is.gd/

curl -s "https://is.gd/create.php?format=simple&url=https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Evolution_of_HTTP" > shorturl.txt
cat shorturl.txt
cat shorturl.txt | xargs curl -Is | grep -i ^Location

Content Negotiation

Example (1): The same content in several different languages

Website: http://www.gnu.org/

curl http://www.gnu.org/ -v -H Accept-Language:de -o gnu.de.html
curl http://www.gnu.org/ -v -H Accept-Language:fr -o gnu.fr.html

Example (2): Redirecting to the mobile version of a website

Website: https://www.youtube.com/

curl https://www.youtube.com/ --http1.1 -v --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1"

See: https://www.whatismybrowser.com/guides/the-latest-user-agent/

Example (3): The same content in several different formats

See: https://www.dbpedia.org/

curl https://dbpedia.org/resource/Grumpy_Cat -v
curl https://dbpedia.org/resource/Grumpy_Cat -v -H Accept:text/html
curl https://dbpedia.org/resource/Grumpy_Cat -v -H Accept:text/html -L -o Grumpy_Cat.html
curl https://dbpedia.org/resource/Grumpy_Cat -v -H Accept:application/json
curl https://dbpedia.org/resource/Grumpy_Cat -v -H Accept:application/json -L -O

Globbing

Example (1)

See: https://www.gnu.org/licenses/

curl https://www.gnu.org/licenses/gpl.\{html,md,txt\} -O
curl "https://www.gnu.org/licenses/gpl.{html,md,txt}" -O
curl "https://www.gnu.org/licenses/gpl.{html,md,txt}" -O -v
curl "https://www.gnu.org/licenses/gpl.{html,md,txt}" --create-dirs -o "licenses/gpl-3.0.#1"

Example (2)

See: https://en.wikisource.org/wiki/The_Hound_of_the_Baskervilles_(Newnes,_1902)

curl https://en.wikisource.org/wiki/The_Hound_of_the_Baskervilles_\(Newnes,_1902\)/Chapter_[1-15] -O
curl https://en.wikisource.org/wiki/The_Hound_of_the_Baskervilles_\(Newnes,_1902\)/Chapter_[1-15] --create-dirs -o "The_Hound_of_the_Baskervilles/Chapter_#1.html"

Multiple Requests

curl https://www.gnu.org/licenses/gpl-3.0.txt -O https://www.apache.org/licenses/LICENSE-2.0.txt -O
curl --head https://dbpedia.org/resource/Hungary https://dbpedia.org/resource/Budapest
curl -H Accept:application/json https://dbpedia.org/resource/Hungary https://dbpedia.org/resource/Budapest -v
curl -H Accept:application/json https://dbpedia.org/resource/Hungary --next -H Accept:application/rdf+xml https://dbpedia.org/resource/Budapest -v

Resume an Interrupted Transfer

See: https://download.bbbike.org/osm/bbbike/Budapest/

Press CTRL + C to interrupt transfer:

curl https://download.bbbike.org/osm/bbbike/Budapest/Budapest.osm.gz -O

Resume transfer with:

curl https://download.bbbike.org/osm/bbbike/Budapest/Budapest.osm.gz -O -C -

Range Requests

See: https://www.gnu.org/licenses/gpl-3.0.txt

The first 100 bytes:

curl https://www.gnu.org/licenses/gpl-3.0.txt -r 0-99
curl https://www.gnu.org/licenses/gpl-3.0.txt -r 0-99 -v

The last 100 bytes:

curl https://www.gnu.org/licenses/gpl-3.0.txt -r -100
curl https://www.gnu.org/licenses/gpl-3.0.txt -r -100 -v

The first and last 100 bytes:

curl https://www.gnu.org/licenses/gpl-3.0.txt -r 0-99,-100
curl https://www.gnu.org/licenses/gpl-3.0.txt -r 0-99,-100 -v

Conditional Requests

See: https://earthquake.usgs.gov/earthquakes/feed/v1.0/csv.php

curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv -O --http1.1 -v
curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv -O --http1.1 -z all_hour.csv -v
sleep 60; curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.csv -O --http1.1 -z all_hour.csv -v

Submitting Form Data

Introduction

Website: https://httpbin.org/

Available options:

• GET method:

  curl --get http://httpbin.org/get -d string=bazinga -d number=42 -v
  curl --get http://httpbin.org/get --data-urlencode "string=Hello, World!" -d number=42 -v
  

• POST method with the application/x-www-form-urlencoded media type:

  curl http://httpbin.org/post -d string=bazinga -d number=42 -v
  curl http://httpbin.org/post -d string=bazinga -d number=42 --trace-ascii -
  

• POST method with the multipart/form-data media type:

  curl http://httpbin.org/post -F string=bazinga -F number=42 -v
  curl http://httpbin.org/post -F string=bazinga -F number=42 --trace-ascii -
  

Example (1): GET

Website: https://blackwells.co.uk/

curl https://blackwells.co.uk/bookshop/search?keyword=sherlock+holmes\&pubDateFrom=2022\&pubDateTo=2023 -o search.html -v
curl --get https://blackwells.co.uk/bookshop/search -d keyword=sherlock+holmes -d pubDateFrom=2022 -d pubDateTo=2023 -o search.html -v
curl --get https://blackwells.co.uk/bookshop/search --data-urlencode keyword="sherlock holmes" -d pubDateFrom=2022 -d pubDateTo=2023 -o search.html -v

Example (2): POST

Website: https://www.base64encode.org/

curl https://www.base64encode.org/ --http1.1 -d input=Hello,\ World! -d charset=UTF-8 -d separator=lf -o output.html --trace-ascii trace.txt -s

Example (3): GET

Website: https://validator.nu/

curl --get https://validator.nu/ --data-urlencode "doc=https://www.w3.org/" -o output.html -v

Example (4): POST

See: https://validator.nu/#file

curl https://www.w3.org/ -o index.html -s
curl https://validator.nu/ -F [email protected] -o output.html --trace-ascii trace.txt -s

Consuming Web Services

Example (1): wttr.in

See: https://wttr.in/, https://github.com/chubin/wttr.in

curl http://wttr.in
curl http://wttr.in/:help
curl http://wttr.in/London
curl http://wttr.in/New+York
curl http://wttr.in/~Tower+Bridge
curl http://wttr.in/~Mount+Everest
curl http://wttr.in/ -H Accept-Language:en
curl http://wttr.in/ -H Accept-Language:de
curl http://wttr.in?lang=hu
curl http://wttr.in?lang=en
curl http://wttr.in?lang=de
curl http://hu.wttr.in
curl http://de.wttr.in
curl http://wttr.in/?format=1
curl http://wttr.in/?format=2
curl http://wttr.in/?format=3
curl http://wttr.in/?format=4
curl http://wttr.in/?format=j1
curl http://wttr.in/Budapest?format=v2
curl http://v2.wttr.in/Budapest
curl http://wttr.in/Moon

Example (2): Nu Html Checker

See: https://validator.nu/ https://github.com/validator/validator/wiki/Service-%C2%BB-Common-params

curl --get https://validator.nu/ --data-urlencode doc=https://whatwg.org/ -d out=json
curl --get https://validator.nu/ --data-urlencode doc=https://whatwg.org/ -d out=xml
curl --get https://validator.w3.org/nu/ --data-urlencode doc=https://www.w3.org/ -d out=xml
curl --get https://validator.w3.org/nu/ --data-urlencode doc=https://www.w3.org/ -d out=xml -d level=error
curl https://whatwg.org/ -o whatwg.html -s
curl https://validator.nu/?out=xml --data-binary @whatwg.html -H "Content-type: text/html; charset=utf-8" -v

Example (3): file sharing (temp.sh)

See: https://temp.sh/

echo "Hello, World!" > hello.txt
curl -F "[email protected]" https://temp.sh/upload

Cookies

See: https://curl.se/docs/http-cookies.html

rm -f cookies.txt
curl https://www.youtube.com/ -c cookies.txt -o /dev/null -v
cat cookies.txt
curl https://www.youtube.com/watch?v=pTBjHjRhx_Y -b cookies.txt -o /dev/null -v

Authentication

Example (1): API key

Website: Rebrickable API

Swagger UI for the API: https://rebrickable.com/api/v3/swagger/

Using the API requires an API key the can be generated here (requires login).

curl https://rebrickable.com/api/v3/lego/sets/60386-1/ -H "Authorization: key <your-api-key>" -v
curl https://rebrickable.com/api/v3/lego/sets/60386-1/minifigs/ -H "Authorization: key <your-api-key>" -v
curl https://rebrickable.com/api/v3/lego/sets/60386-1/parts/ -H "Authorization: key <your-api-key>" -v

Example (2): Basic Authentication

Website: GitHub

curl https://api.github.com/user -v
curl https://api.github.com/user -v -u <username>

See:

• https://docs.github.com/en/rest
• https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens

Fun and Entertainment

VT100 animation

Website: http://artscene.textfiles.com/vt100/

curl -s http://artscene.textfiles.com/vt100/globe.vt | pv -L4K -q

ASCII animation

Website: http://ascii.live/ https://github.com/hugomd/ascii-live

curl http://ascii.live/forrest
curl ascii.live/list
curl ascii.live/rick
curl ascii.live/donut

Impersonating a Browser

Some websites do not like non-browser user agents, and will send, e.g., 403 (Forbidden) responses for requests sent by curl. In this, case, you can try curl-impersonate, a special build of curl that can impersonate major browsers, e.g., Firefox or Chrome.

For example, the command

curl --head https://www.nasdaq.com/

terminates with an error.

To trick the server, use the command

curl_chrome116 --head https://www.nasdaq.com/

instead, that impersonates the Chrome browser.

Note that curl-impersonate is available for Linux and macOS systems.

You can read about how websites can block non-browser user agents here:

• TLS fingerprinting
• HTTP/2 fingerprinting

Unfortunately, curl-impersonate is not always we able to trick websites. For example, it is not possible to access Netflix with curl-impersonate. Thus, e.g.,

curl_chrome116 --head https://www.netflix.com/en/

results in a 403 (Forbidden) response.

In many cases, the problem is caused by Cloudflare that hosts the website. The Browser Integrity Check feature of Cloudflare is provided to block non-browser user agents that is enabled by default.