Pandoc, YAML, JSON, CSV, Sqlite

README.md

Pandoc, YAML, JSON, CSV, Sqlite

Use Pandoc to format data.

Self-contained Pandoc document

The document.md file contains data in its metadata YAML block and is its own template:

$ pandoc --template document.md document.md
---
title: The document title
host:
- name: host-1
  ip: 172.17.0.1
- name: host-2
  ip: 172.17.0.2
---

Some hosts:

- host-1: 172.17.0.1
- host-2: 172.17.0.2

The result is a normal Markdown document that can be processed with Pandoc as usual.

Separate data

The data can also come from an external file:

$ pandoc --template document.md hosts.yml document.md
---
title: The document title
host:
- name: host-1
  ip: 172.17.0.1
- name: host-2
  ip: 172.17.0.2
---

Some hosts:

- host-1: 172.17.0.1
- host-2: 172.17.0.2
- host-3: 172.17.0.3

Note: here the hosts key in the metadata provided by document.md is not used because Pandoc is left-biased when merging multiple metadata blocks.

Other formats

Several scripts are provided to change data format. For instance:

• From hosts.json (one object per line) to CSV,
• From CSV to a Sqlite database,
• From a Sqlite database to JSON again (a list of objects),
• From a Sqlite database to YAML (using the previous script).

csv-to-sqlite.sh

#! /usr/bin/env bash

sqlite3 hosts.db -cmd '.mode csv' '.import hosts.csv hosts'

# Demonstrate the import succeeded
sqlite3 hosts.db '.schema hosts'
sqlite3 hosts.db 'select * from hosts'

document.md

title	host
The document title	name ip host-1 172.17.0.1 name ip host-2 172.17.0.2

Some hosts:

$for(host)$

• $host.name$: $host.ip$ $endfor$

hosts.json

{ "name":"host-1", "ip":"172.17.0.1" }
{ "name":"host-2", "ip":"172.17.0.2" }
{ "name":"host-3", "ip":"172.17.0.3" }

json-to-csv.sh

#! /usr/bin/env bash

# Construct the CSV header
head -1 hosts.json | jq -r 'keys | @csv' > hosts.csv

# Extract rows
cat hosts.json | jq -r '[.[]] | @csv' >> hosts.csv

sqlite-to-json.sh

#! /usr/bin/env bash

sqlite3 hosts.db \
  "select json_group_array(json_object('ip', ip, 'name', name)) as host
  from (select * from hosts)" | jq .

sqlite-to-yaml.sh

#! /usr/bin/env bash

echo '---'
echo host:
./sqlite-to-json.sh | jq -r -f to-yaml.jq
echo '---'

to-yaml.jq

def y:
  (objects | to_entries[] | (.value | type) as $type |
    if $type == "array" then
      "\(.key):", (.value | y)
    elif $type == "object" then
      "\(.key):", "  \(.value | y)"
    else
      "\(.key): \(.value)"
    end
  )
  // (arrays | select(length > 0)[] | [y] |
    "  - \(.[0])", "    \(.[1:][])"
  )
  // .
  ;
y