How to use pandoc with gitlab markdown

readme.md

Table of Contents

• How to use pandoc with gitlab markdown
  • Installing the Dependencies
  • Setting Up the Environment
  • Generate PDF/HTML

How to use pandoc with gitlab markdown

If you're like me, then you use gitlab-flavored markdown quite a bit. The combination of mermaid and plantuml graphs and katex math really makes for a productive experience. However, due to issues like gitlab-org/gitlab#215084 and gitlab-org/gitlab#243533, you probably also want to have some other way to render your notes, rather than relying on gitlab to do it for you.

This guide will show you how to do just that with a tool called pandoc, as long as you're OK with not having all the features of gitlab markdown. At the moment, this setup supports the following features:

• Mermaid Graphs
• PlantUML Graphs
• Katex Math
• Relative and Absolute References to Project Files
• References to Directories with READMEs

This setup does run in CI/CD, which is how I primarily use it, but this guide will focus on how to set this up on your own machine. If you're interested in a CI/CD project template though, I have one on my gitlab. I will tend to update that faster than this document, and I track known issues with it there too.

Installing the Dependencies

To produce this setup, I installed:

Dependency	Rationale
mermaid-cli	Required by timofurrer's pandoc-mermaid-filter
plantuml	Required by timofurrer's pandoc-plantuml-filter
texlive-full[1]	Required by the pandoc LaTeX template
My fork of @timofurrer's pandoc-plantuml-filter	Allows pandoc to render plantuml code blocks as graphs
My fork of @timofurrer's pandoc-mermaid-filter	Allow pandoc to make sense of mermaid code blocks as graphs
This GitHub pandoc html5 template	Used to make the html all pretty :D
This pandoc LaTeX Template	Used to make the pdf all pretty :D
@lierdakil's gitlab-math.lua script	Supports gitlab's special math syntax.
My fix-links.lua script	Fixes links and urls to resources within the project.

... and of course, pandoc itself!

I'm on Manjaro Linux, so I ended up using the following commands to install these:

# install plantuml & mermaid
yarn global add @mermaid-js/mermaid-cli
pamac install plantuml
# Install pandoc plantuml & mermaid filter
pip install git+https://github.com/MyriaCore/pandoc-plantuml-filter.git
pip install git+https://github.com/MyriaCore/pandoc-mermaid-filter.git
# Install lua filters
mkdir -p ~/.pandoc/filters
wget https://gist.githubusercontent.com/lierdakil/00d8143465a488e0b854a3b4bf355cf6/raw/gitlab-math.lua
mv gitlab-math.lua ~/.pandoc/filters/gitlab-math.lua
wget https://gist.githubusercontent.com/MyriaCore/75729707404cba1c0de89cc03b7a6adf/raw/fix-links.lua
mv fix-links.lua ~/.pandoc/filters/fix-links.lua

Just a quick note, you will need to install yarn to install mermaid-cli, unfortunately. You could try npm, but that didn't work as well for me.

---

1: On manjaro, I had to get it from the aur, but if you're on an ubuntu-based distro, you should be able to get it more easily. ↩

Setting Up the Environment

All you'll need to do to setup the environment is expose the mermaid binary (assuming you installed via yarn):

# Exposes mmdc (mermaid binary) for us
export PATH=$(yarn global bin):$PATH 

Generate PDF/HTML

I wrote a small script to generate a pdf or html file (at the user's request). If you don't care about the commands and just want to get up and running, I'd recommend putting it somewhere on your $PATH:

wget https://gist.githubusercontent.com/MyriaCore/75729707404cba1c0de89cc03b7a6adf/raw/570ec66358b95f634a28be69ba5de59854e41b24/gfm.sh
mv gfm.sh /usr/local/bin

# create readme.html
gfm.sh readme.md
gfm.sh -html readme.md

# create readme.pdf
gfm.sh -pdf readme.md

This script basically just provides a nice wrapper for the commands below. If you're interested in understanding those, keep reading. Otherwise, gfm.sh should complete your setup.

HTML Conversion

Using the Github html5 template:

MERMAID_THEME='neutral' # use gitlab's mermaid theme
pandoc --from markdown --to html5   \
   --self-contained --standalone    \
   --filter pandoc-plantuml         \
   --filter pandoc-mermaid          \
   --lua-filter gitlab-math.lua     \
   --lua-filter fix-links.lua       \
   --katex --template=GitHub.html5  \
   -o <name of output>.html <name of input>.md

The --self-contained option allows links to images, pictures, and mermaid/plantuml graphs to be embedded with data:... URIs instead of relative links. This allows the HTML file to be truly portable, if not a bit large.

NOTE: Using --self-contained did work locally on my laptop, but didn't work with my CI/CD setup. I had to remove this flag in the makefile as a result. See this discussion for more info.

PDF Conversion

The best template I could find out there is Eisvogel, which is a latex template that will help us create a pdf.

MERMAID_THEME='neutral' # use gitlab's mermaid theme
pandoc --from markdown --to latex  \
  --self-contained --standalone    \
  --filter pandoc-plantuml         \
  --filter pandoc-mermaid          \
  --lua-filter gitlab-math.lua     \
  --lua-filter fix-links.lua       \
  --katex --template=eisvogel      \
  -o <name of output>.pdf <name of input>.md

In my experience, this template is super buggy in combination with pandoc-mermaid-filter and pandoc-plantuml-filter. The images pretty regularly spill over the margins of the document, even when using svg images. I'll look into this further to see if I can tweak it, or find a better template.

NOTE: Using --self-contained did work locally on my laptop, but didn't work with my CI/CD setup. I had to remove this flag in the makefile as a result. See this discussion for more info.

fix-links.lua

-- see if the file exists
function file_exists(file)
  local f = io.open(file, "rb")
  if f then f:close() end
  return f ~= nil
end

-- 1. Replaces .md with .html
-- 2. Replaces absolute paths with relative ones
-- 3. Replaces folder links with links to their index / readme
function fix_link (url) 
  -- Replace md with html
  fixed_url = url:gsub("%.md", ".html")
  -- Replace project-root-relative (i.e. /path/to/thing.md) paths with relative ones
  fixed_url = fixed_url:gsub("^/", (
    function(s)
      basedir = io.popen("realpath --relative-to=\"$(dirname \"" .. s:sub(2) .. "\")\" .", 'r'):read('*a')
      return basedir:gsub("%s", "") .. "/" .. s:sub(2)
    end
  ))
  -- Allowed readme / index names
  dir_name = {} -- values are functions that return the path to the new link
  dir_name["index.md"] = (function(s) return s:gsub("/?$", "/index.html") end)
  dir_name["INDEX.md"] = (function(s) return s:gsub("/?$", "/INDEX.html") end)
  dir_name["readme.md"] = (function(s) return s:gsub("/?$", "/readme.html") end)
  dir_name["README.md"] = (function(s) return s:gsub("/?$", "/README.html") end)
  -- Is the url pointing to a folder?
  file_or_folder = io.popen("[ -d '" .. fixed_url .. "' ] && echo 'folder'", 'r'):read('*a')
  -- If it's a folder, find the readme / index and replace the link with that. 
  if file_or_folder:gsub("%s", "") == "folder" then
    for index,mk_path in pairs(dir_name) do
      if file_exists(fixed_url .. "/" .. index) then
        fixed_url = mk_path(fixed_url)
        break
      end
    end
  end
  return fixed_url
end


function Link(link) link.target = fix_link(link.target); return link end
function Image(img) img.src = fix_link(img.src); return src end
-- return {{Meta = Meta}, {Link = Link, Image = Image}}

gfm.sh

#!/bin/bash

if [[ $# == 0 ]] || [[ $1 == '-h' ]] || [[ $1 == '--help' ]]; then
  cat <<-EOF
	USAGE: gfm.sh [ OPTIONS ] <markdown file>
	
	Converts Gitlab Flavored Markdown into PDF or HTML format
	
	DEFAULT OPTIONS: gfm.sh -html --file <markdown file>
	
	FORMAT OPTIONS:
	  -pdf           Converts the GFM to pdf
	  -html          Converts the GFM to html
	
	OUTPUT OPTIONS:
	  -p --print     Prints the generated output     
	  -f --file      Creates a file of the same name
	EOF
  exit 0
fi

# Set defaults
filetype='html'
output='file'

# Process Arguments
let "n = $#"
while [[ $n > 1 ]]; do
  echo $n
  case $1 in
    -pdf|-html) filetype="${1/-/}" ;;
    -p|--print|-f|--file) 
      output="${1/-/}" ;;
    --)
      shift 1
      break ;;
  esac
  let "n--"
  shift 1
done

# Run script
if [[ $output == 'file' ]]; then
  case $filetype in
    html)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to html5   \
         --self-contained --standalone    \
         --filter pandoc-plantuml         \
         --filter pandoc-mermaid          \
         --lua-filter gitlab-math.lua     \
         --lua-filter fix-links.lua       \
         --katex --template=GitHub.html5  \
         -o "${1/.md/.html}" "$1"
      ;;
    pdf)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to latex  \
        --self-contained --standalone    \
        --filter pandoc-plantuml         \
        --filter pandoc-mermaid          \
        --lua-filter gitlab-math.lua     \
        --lua-filter fix-links.lua       \
        --katex --template=eisvogel      \
        -o "${1/.md/.pdf}" "$1"
      ;;
  esac
else
  case $filetype in
    html)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to html5   \
         --self-contained --standalone    \
         --filter pandoc-plantuml         \
         --filter pandoc-mermaid          \
         --lua-filter gitlab-math.lua     \
         --lua-filter fix-links.lua       \
         --katex --template=GitHub.html5  \
         "$1"
         ;;
    pdf)
      MERMAID_THEME='neutral'
      pandoc --from markdown --to latex  \
        --self-contained --standalone    \
        --filter pandoc-plantuml         \
        --filter pandoc-mermaid          \
        --lua-filter gitlab-math.lua     \
        --lua-filter fix-links.lua       \
        --katex --template=eisvogel      \
        "$1"
        ;;
  esac
fi