jq cheatsheet

jq-cheatsheet.md

JQ Cheatsheet

Sample JSON 1:

{
  "data": [
    {
      "bar": "blah",
      "quux": "garply",
      "foo": [
        {
          "x": 3,
          "y": 1,
          "z": 1
        },
        {
          "x": 5,
          "y": 3,
          "z": 1
        }
      ]
    },
    {
      "bar": "asdf",
      "quux": "corge",
      "foo": [
        {
          "x": 2,
          "y": 3,
          "z": 1
        }
      ]
    },
    {
      "bar": "baz",
      "quux": "garply",
      "foo": [
        {
          "x": 5,
          "y": 2,
          "z": 1
        },
        {
          "x": 3,
          "y": 1,
          "z": 1
        }
      ]
    },
    {
      "bar": "hello",
      "quux": "corge"
    }
  ]
}

Patterns and idioms

Filtering parent objects on properties of child arrays

# sample json 1
[
  .data[]
  | select(.foo != null)
  | {
    bar,
    foo: [.foo[] | {x, y}] | sort_by(.x)
       # ^               ^   (the brackets are critical)
  }
  | select(any(.foo[]; .x > 3 and .y > 2)) # <-- (any is critical)
]
| sort_by(.bar)

Manipulating child arrays of property

# sample json 1
.data |= map(select(.quux == "garply"))

Merging counts by key in child array

# sample json 1
# https://stackoverflow.com/questions/28484534/how-do-i-sum-the-values-in-an-array-of-maps-in-jq
[
  .data[].foo
  | select(. != null)
  | map(to_entries)
  | flatten 
  | reduce .[] as $e ({}; .[$e.key] += $e.value)
]
| map(to_entries)
| add
| group_by(.key) # <-- important
| map({key: .[0].key, value: map(.value) | add})
| sort_by(-.value)
| from_entries

Attach parent property to every nested child object and flatten

[.data[] as $parent | $parent.foo[]? | . + {bar: $parent.bar}]

String comparison/searching

def str_case_eq(s): (. | ascii_upcase) == (s | ascii_upcase);
def str_case_contains(s): (. | ascii_upcase) | index(s | ascii_upcase); # see also 'contains' instead of 'index'

"hello" as $foo
| [
  ($foo | str_case_eq("HEllO")),       # true
  ($foo | str_case_eq("EllO")),        # false
  ($foo | str_case_contains("EllO")),  # 1 (0 is also truthy!)
  ($foo | str_case_contains("zEllO"))  # null 
]

Truncating all string fields

jq 'walk(if type == "string" then .[:50] else . end)' foo.json

Convert array of objects to CSV

[
  {
    "accountId": "x",
    "accountNumber": "y",
    "nickname": "z"
  },
  {
    "accountId": "xx",
    "accountNumber": "yy",
    "nickname": "zz"
  }
]

jq -r '(["accountId", "accountNumber", "nickname"], (.[] | [.accountId, .accountNumber, .nickname])) | @csv' input.json

Or generalize with this answer

Big dump of unorganized links

• merging objects
  • https://stackoverflow.com/questions/28484534/how-do-i-sum-the-values-in-an-array-of-maps-in-jq
  • https://stackoverflow.com/questions/34477547/how-to-combine-the-sequence-of-objects-in-jq-into-one-object
  • https://stackoverflow.com/questions/51491317/jq-how-to-merge-multiple-objects-into-one
  • https://gist.github.com/pkoppstein/5eac78d3e9618cfabd7d
  • https://stackoverflow.com/questions/49573125/jq-how-to-combine-disjoint-object-values-as-a-single-object-of-key-value-pairs
  • https://stackoverflow.com/questions/54956517/use-jq-to-combine-two-arrays-of-objects-on-a-certain-key
  • https://unix.stackexchange.com/questions/588063/merging-multiple-json-file-into-one-object-using-jq-with-argument
  • https://unix.stackexchange.com/questions/588098/merging-multiple-json-files-removing-duplicate-objects-by-a-field
  • jqlang/jq#1016
• https://stackoverflow.com/questions/52065570/how-do-i-sum-all-numbers-from-output-of-jq
• https://unix.stackexchange.com/questions/610461/how-to-merge-arrays-from-multiple-json-files-with-jq
• https://stackoverflow.com/questions/34477547/how-to-combine-the-sequence-of-objects-in-jq-into-one-object
• https://stackoverflow.com/questions/18592173/select-objects-based-on-value-of-variable-in-object-using-jq
• https://stackoverflow.com/questions/68411502/jq-add-variable-property-to-existing-object
• https://stackoverflow.com/questions/68159673/jq-case-insensitive-contains
• https://stackoverflow.com/questions/51718102/filter-json-via-bash-case-insensitive
• https://stackoverflow.com/questions/39971324/using-jq-to-select-a-parent-by-searching-child-key-value-pair
• jqlang/jq#684
• jqlang/jq#738
• https://remysharp.com/drafts/jq-recipes
• https://stackoverflow.com/questions/41045659/merge-two-arrays-into-a-single-object-with-jq
• https://stackoverflow.com/questions/53420046/summarize-multiple-values-with-jq-reduce
• jqlang/jq#1646
• https://stackoverflow.com/questions/51970546/using-jq-select-multiple-keys-and-return-them-in-an-array
• https://stackoverflow.com/questions/54187750/jq-lifting-fields-from-array-of-objects-into-the-parent-object
• https://stackoverflow.com/questions/47887535/filtering-objects-in-jq-by-existence-of-nested-array-values
• https://stackoverflow.com/questions/46254655/how-to-merge-json-objects-by-grouping-on-key-with-jq
• https://stackoverflow.com/questions/46397592/jq-removing-duplicates-using-unique-by
• https://stackoverflow.com/questions/18592173/select-objects-based-on-value-of-variable-in-object-using-jq/31911811#31911811
• https://stackoverflow.com/questions/26701538/how-to-filter-an-array-of-objects-based-on-values-in-an-inner-array-with-jq
• https://stackoverflow.com/questions/43221453/jq-group-and-key-by-property
• https://stackoverflow.com/questions/48843494/how-to-convert-objects-to-array-with-jq
• https://stackoverflow.com/questions/43259563/how-to-check-if-element-exists-in-array-with-jq
• jqlang/jq#1056
• https://stackoverflow.com/questions/53594410/jq-transform-json-child-array-objects-to-delimited-string
• https://stackoverflow.com/questions/47894527/jq-flattening-json-arrays
• https://stackoverflow.com/questions/46460631/how-can-i-count-number-of-elements-after-query
• https://stackoverflow.com/questions/58881716/generate-field-value-frequency-count-with-jq
• https://stackoverflow.com/questions/63908137/jq-how-to-merge-multiple-arrays
• https://unix.stackexchange.com/questions/370306/split-a-string-field-into-an-array-in-jq
• https://stackoverflow.com/questions/44526740/how-to-trim-white-space-for-every-element-in-jq
• https://stackoverflow.com/questions/25649960/how-to-select-items-in-jq-based-on-value-in-array
• https://stackoverflow.com/questions/40027395/passing-bash-variable-to-jq
• https://stackoverflow.com/questions/34745451/passing-arguments-to-jq-filter
• https://stackoverflow.com/questions/30331504/how-to-sort-a-json-file-by-keys-and-values-of-those-keys-in-jq
• https://stackoverflow.com/questions/43259563/how-to-check-if-element-exists-in-array-with-jq
• https://stackoverflow.com/questions/25649960/how-to-select-items-in-jq-based-on-value-in-array
• https://stackoverflow.com/questions/38314400/jq-remove-object-from-multiple-arrays
• https://serverfault.com/questions/991982/jq-get-values-from-children-array-and-display-on-parent-array
• https://stackoverflow.com/questions/44526740/how-to-trim-white-space-for-every-element-in-jq
• https://programminghistorian.org/en/lessons/json-and-jq#filter-select
• https://stackoverflow.com/questions/50864894/how-to-filter-keys-by-name-then-access-nested-object-using-jq
• https://stackoverflow.com/questions/45165303/how-to-combine-an-array-into-a-single-string-value-when-using-csv-output-in-jq
• https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4