jq: JSONL ↔︎ JSON conversion

jq_jsonl_conversion.md

jq: JSONL ↔︎ JSON conversion

Prerequisites

• jq — https://jqlang.github.io/jq/ — "like sed for JSON data"

  There are several options available for installing jq. I prefer to use Homebrew: brew install jq

1. JSONL → JSON

   jq -s '.' input.jsonl > output.json

2. JSON → JSONL

   jq -c '.[]' input.json > output.jsonl

It's unbelievable, jq can solve this in one line. I was about to embark on writing yet-another-python-script.py to convert a gzipped nested JSON file to JSONL, but thankfully I came across this post.

Suppose you have a JSON like this:

{
   "meta" : { }

 , "data" : [
    { "idx" : 1
      , "input" : "ABC"
     , target : "123"
     , some_other_field : "zzz" 
    },

   { "idx" : 2
      , "input" : "DEF"
     , target : "456"
     , some_other_field : "zzz" 
    }, 
   ...

 ]
}

You can use the following one-line command to extract the 'data' array, keep the 'input' and 'target' fields only, and generate JSONL:
gunzip -c somefile.json.gz | jq .data | jq -c '.[] | {input, target}'

Output

{"input" : "ABC, "target" : "123"}
{"input" : "DEF, "target" : "456"}
...