Tips for making JSON API calls with curl and jq: pretty-printing and paging responses, sending JSON POSTs with --json @-, embedding text like SQL queries with jq -Rs, and a jqless pager function.

curl-jq-api-tips.md

Lately I've been making a LOT of API calls with curl, mostly to PostHog's API, and picked up some tricks along the way.

Let's start by trying this:

curl --no-progress-meter --dump-header=% https://xkcd.com/info.0.json | jq .

When APIs return JSON a lot of them don't pretty-print it. Pipe it into jq ., install it if you don't have it, and you'll get the JSON back with both pretty-printing and syntax highlighting.

--no-progress-meter gets rid of that horrible progress meter which just gets in the way of fast API calls.

--dump-header=% works almost exactly like -i does, printing out the HTTP response headers, except it does so to stderr where it will not cause jq to error by attempting to interpret it as JSON.

But that's not enough. Sometimes an API call can be just so big that it fills up your terminal. But if you pipe it into less then you lose all the syntax highlighting. That's where this comes in.

curl --no-progress-meter --dump-header=% https://jsonplaceholder.typicode.com/photos | jq --color-output . - | less -R -

This became so common for me that I permanently embedded this shortcut in my .bashrc:

jqless () 
{ 
    jq --color-output "${@:-.}" - | less -R -x2 --quit-if-one-screen --no-init --file-size --wordwrap --mouse --wheel-lines=4 -
}

Try these now:

curl --no-progress-meter --dump-header=% https://jsonplaceholder.typicode.com/photos | jqless
curl --no-progress-meter https://xkcd.com/info.0.json | jqless -r '.title + ":\n" + .img'

Pretty printed. Syntax highlighted. less is skipped if the API response is short.

Now, this is all great for reading JSON from API calls, but in practice API calls often need you to include permission headers and write your own JSON.

Knowing --oauth2-bearer="$SOME_API_KEY" exists is great, it's much shorter than typing out a full -H "Authorization: Bearer ..." header every time, even when you're not using OAuth.

And a bash one-liner to get that API key into $SOME_API_KEY without it landing in your terminal history:

unset SOME_API_KEY; read -rsp "Some API key: " SOME_API_KEY; echo '';

Now for sending JSON your new best friend is --json @-. This short little thing defaults the request to a POST, sends a JSON Accept header, a JSON Content-Type, and reads the POST body from stdin. From there all you need to do is pipe some JSON in from jq and you can easily write any complex JSON you want to send in an API request.

echo '{"Now": "trying to write json", "with": ["all", "these", "quotation marks"]}' is a pain.

jq -n '{a: 1, launch: [3, 2, 1]}' | curl --no-progress-meter https://httpbin.org/post --json @- | jqless will let you write js-like syntax in jq's "filter" that will turn into valid JSON.

Sometimes you need to embed a complex block of text into your JSON. I needed to do this a lot when sending SQL-like queries to PostHog. Fortunately -Rs will read stdin into a multi-line string instead of interpreting it as json.

echo -e '<p>naïve estimates — 80% done</p>\n<em>≠ 80% done</em>' | jq -Rs --arg t "AI quote of the day" '{userId: 1, title: $t, body: .}' | curl --no-progress-meter https://jsonplaceholder.typicode.com/posts --json @- | jqless

Now if that is not enough and you need lots of newlines and symbols in your text or a lot of options in your jq filter then cat query.text | jq -Rsf filter.jq | ... will let you write the block of text in a .txt file (or .sql or whatever else with syntax highlighting in your IDE) and the filter syntax in the .jq file then run it together.

If this keeps up I might end up writing something to shorten all the args I need to pass to curl for JSON APIs too. What do you think?