Processing JSON using jq

jq is useful to slice, filter, map and transform structured json data.

Installing jq

On Ubuntu

sudo apt-get install jq

On Mac OS

brew install jq

On AWS Linux

Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/

Installing from the source proved to be tricky.

Useful arguments

When running jq, the following arguments may become handy:

Argument	Description
`--version`	Output the jq version and exit with zero.
`--sort-keys`	Output the fields of each object with the keys in sorted order.

Basic concepts

The syntax for jq is pretty coherent:

Syntax	Description
,	Filters separated by a comma will produce multiple independent outputs
?	Will ignore error if the type is unexpected
[]	Array construction
{}	Object construction
+	Concatenate or Add
-	Difference of sets or Subtract
length	Size of selected element
\|	Pipes are used to chain inpiuts and outputs in a similar fashion to bash

Dealing with json objects

Description	Command
Display all keys	`jq 'keys'`
Adds + 1 to all items	`jq 'map_values(.+1)'`
Delete a key	`jq 'del(.foo)'`
Convert an object to array	`to_entries \| map([.key, .value])`

Dealing with fields

Description	Command
Concatenate two fields	`fieldNew=.field1+' '+.field2`

Dealing with json arrays

Slicing and Filtering

Description	Command
All	`jq .[]`
First	`jq '.[0]'`
Range	`jq '.[2:4]'`
First 3	`jq '.[:3]'`
Last 2	`jq '.[-2:]'`
Before Last	`jq '.[-2]'`
Select where value is `foo` or `bar`	`q 'select(.value == "foo" or .value == "bar")'`
Select array of int by value	`jq 'map(select(. >= 2))'`
Select array of objects by value	`jq '.[]` \| `select(.id == "second")'`
Select by type	`jq '.[]` \| `numbers'` with type being arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description	Command
Add + 1 to all items	`jq 'map(.+1)'`
Delete 2 items	`jq 'del(.[1, 2])'`
Concatenate arrays	`jq 'add'`
Flatten an array	`jq 'flatten'`
Create a range of numbers	`jq '[range(2;4)]'`
Display the type of each item	`jq 'map(type)'`
Sort an array of basic type	`jq 'sort'`
Sort an array of objects	`jq 'sort_by(.foo)'`
Group by a key - opposite to flatten	`jq 'group_by(.foo)'`
Minimun value of an array	`jq 'min'` .See also `min`, `max`, `min_by(path_exp)`, `max_by(path_exp)`
Remove duplicates	`jq 'unique'` or `jq 'unique_by(.foo)'` or `jq 'unique_by(length)'`
Reverse an array	`jq 'reverse'`
Convert `{id: id-val, value: val}` to a json	`'[ .path` \| `{ (.id`\|`tostring): .value }]'`

Use select(boolean_expression)

          jq '.[] | select(.id == "second")'
Input:	  [{"id": "first", "val": 1}, {"id": "second", "val": 2}]
Output: 	{"id": "second", "val": 2}

        jq 'map(select(. >= 2))'
Input	  [1,5,3,0,7]
Output 	[5,3,7]

Create csv file

echo "ID,NAME,TYPE" > output.csv cat input.json | jq -r '.resource_list[] | [.id, .name, .type] | @csv' >> output.csv

Examples

Find length of a list cat users-list.json | jq -r '.results | length'
To extract top level attributes "timestamp" and "report" jq '. | {timestamp,report}'
To extract name and age of each "results" item jq '.results[] | {name, age}'
To extract name and age as text values instead of JSON jq -r '.results[] | {name, age} | join(" ")'

Filter this by attribute

   jq '.results[] | select(.name == "John") | {age}'          # Get age for 'John'
   jq '.results[] | select((.name == "Joe") and (.age = 10))' # Get complete records for all 'Joe' aged 10
   jq '.results[] | select(.name | contains("Jo"))'           # Get complete records for all names with 'Jo'
   jq '.results[] | select(.name | test("Joe\s+Smith"))'      # Get complete records for all names matching PCRE regex 'Joe\+Smith'

Avoid null output when accessing non-existing keys jq '.mykey | select(. != null)'
If you want to combine subkeys at different levels jq '.items[] | { "created" : .metadata["created"], name }' OR jq '.items[] | .metadata["created"], .name'
If you want to extract all properties into json jq '. | to_entries[] | select( .key | contains("prop"))' This will output each key value pair as: {"key": "Keyname", "value": "Value"}
When you just want the keys jq '.animals | keys'
If you want to convert JSON of format from (8) to a proper key value json '[ .path | { (.key|tostring): .value }]'
Match a regular expression .path | match("([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+.[a-zA-Z0-9_-]+)"; "i") | .string will search for emails .path | match("\\d+") | .string will search for digits
Replace strings .email | sub("john";"richard";"i")

Changing values with jq

Merging/overwriting keys

echo '{ "a": 1, "b": 2 }' |\
jq '. |= . + {
  "c": 3
}'

Adding elements to lists

echo '{ "names": ["Marie", "Sophie"] }' |\
jq '.names |= .+ [
   "Natalie"
]'

Delete values with jq

jq 'del(.somekey)' input.json

Merge JSON strings

For example merge three object lists:

echo '[ {"a":1}, {"b":2} ]' | \
jq --argjson input1 '[ { "c":3 } ]' \
   --argjson input2 '[ { "d":4 }, { "e": 5} ]' \
   '. = $input1 + . +  $input2'

Merge files (since jq 1.4)

The following command will merge "somekey" from both passed files

jq -s '.[0] * .[1] | {somekey: .somekey}' <file1> <file2>

Handle Empty Arrays

When you want to iterate over an array, and the array you access is empty you get something like

jq: error (at <stdin>:3): Cannot iterate over null (null)

To workaround the optional array protect the access with

select(.my_array | length > 0)

Testing Types

$ echo '[true, null, 42, "hello", []]' | ./jq 'map(type)'
["boolean","null","number","string","array"]

Using it in shell scripts

Parsing JSON into env vars

To fill environment variables from JSON object keys (e.g. $FOO from jq query ".foo")

export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"')

To make a bash array

read -a bash_array < <(jq -r .|arrays|select(.!=null)|@tsv)

JSON template using env vars

To create proper JSON from a shell script and properly escape variables:

jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'

URL Encode

Quick easy way to url encode something

date | jq -sRr @uri

String Concat

Concatenation like this:

echo '{ "object" : { "name": "banana", "color": "yellow" }}' |\
jq -r '.object | (.name)+" is "+(.color)'

will print banana is yellow.

String Interpolation

Or using Interpolation:

echo '{ "object" : { "name": "banana", "color": "yellow" }}' |\
jq -r '.object | "\(.name) is \(.color)"'

will also print banana is yellow.

Turn A List From A Command Into JSON

There are a lot of command-line utilities that produce a list of things. Since JSON is a universal data format, it would be useful to be able to quickly turn some items from stdout into a JSON list.

The jq utility can help with this.

Let's say I'm working with the following git command that lists changed files in a specific directory.

$ git diff --name-only | grep some/dir

I can then pipe that list of files to jq with a few flags.

$ git diff --name-only \
  | grep some/dir \
  | jq -R -s 'split("\n")[:-1]'

Here's what is going on:

The -R flag tells jq to accept raw input, rather than looking for JSON.
The -s flag is short for --slurp and tells jq to read in the entire input before applying the filter.
The string argument is the filter to be applied to the output. It splits on newlines and then takes the entire array except for the last item ([:-1]) which would be an empty string for the trailing newline.
jq automatically turns the whole thing into a formatted JSON list.

References

Reference - https://stedolan.github.io/jq/manual/

bhattisatish/jq-cheetsheet.md