Skip to content

Instantly share code, notes, and snippets.

@eevmanu

eevmanu/pandoc_markitdown_anydoc_comparison.md

Created August 4, 2026 20:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save eevmanu/5da8c1919bbcc0d459dc82bdea04c93f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save eevmanu/5da8c1919bbcc0d459dc82bdea04c93f to your computer and use it in GitHub Desktop.
Download ZIP
Comprehensive Technical Comparison: Pandoc vs. MarkItDown vs. AnyDoc
Raw
pandoc_markitdown_anydoc_comparison.md

Comprehensive Technical Comparison: Pandoc vs. MarkItDown vs. AnyDoc

Executive Summary

Document conversion is a foundational capability across enterprise search, Knowledge Retrieval-Augmented Generation (RAG) pipelines, publishing systems, and multi-modal AI applications. This authoritative technical report presents a deep, source-code grounded architectural evaluation of three premier document transformation technologies:

  1. Pandoc (v3.10.1): A pure functional document compiler written in Haskell, built around a universal Abstract Syntax Tree (AST) designed for bidirectional multi-target publishing.
  2. MarkItDown: A Python-native object converter pipeline created by Microsoft, designed for flexible, multi-modal ingestion (including LLM vision and audio transcription) targeting Markdown.
  3. AnyDoc (v0.1.3): A ultra-fast, zero-dependency document parsing engine written in modern Rust, engineered specifically by Firecrawl for high-throughput LLM-ready Markdown generation in single-digit milliseconds.

1. Architectural Overview & Core Design Philosophy

1.1 Pandoc: Functional AST Compiler

Input Document ---> Reader [Text.Pandoc.Readers.*] 
                   ---> Pandoc AST [pandoc-types: Text.Pandoc.Definition]
                   ---> AST Filters [Text.Pandoc.Filter / pandoc-lua-engine]
                   ---> Writer [Text.Pandoc.Writers.*] 
                   ---> Output Document

Pandoc operates as a classical multi-stage language compiler. Its core design philosophy centers on the Universal Canonical Intermediate Representation (AST) defined in the pandoc-types library (Text.Pandoc.Definition).

  • Core Haskell Types:
    • Pandoc: The root node wrapping document metadata and block sequences: data Pandoc = Pandoc Meta [Block].
    • Block Enum: Represents structural document blocks including Para [Inline], Header Int Attr [Inline], CodeBlock Attr Text, BlockQuote [Block], OrderedList ListAttributes [[Block]], BulletList [[Block]], Table Attr Caption [ColSpec] TableHead [TableBody] TableFoot, Figure Attr Caption [Block], and Div Attr [Block].
    • Inline Enum: Represents inline markup elements including Str Text, Emph [Inline], Strong [Inline], Link Attr [Inline] Target, Image Attr [Inline] Target, Math MathType Text, Code Attr Text, and Note [Block].
  • Modular Monadic Decoupling:
    • Readers (Text.Pandoc.Readers.*): Monadic parsers (utilizing Parsec/Megaparsec and XML parsers) that transform source byte streams or text into Pandoc AST.
    • Writers (Text.Pandoc.Writers.*): Layout generators utilizing the doclayout (Doc) string builder monad to serialize the Pandoc AST into output formats.
    • Filters (Text.Pandoc.Filter): Programmatic AST transformers. JSON filters execute as separate subprocesses receiving JSON-serialized AST over standard I/O, whereas Lua filters execute in-process via pandoc-lua-engine using the hslua C-Lua embedded runtime for zero-IPC traversal (walk, walkM).
    • Server Component: pandoc-server exposes a REST microservice interface wrapping the internal Haskell library.

1.2 MarkItDown: Python Dynamic Converter Pipeline

Input Stream ---> MarkItDown Engine [markitdown/_markitdown.py]
              ---> StreamInfo Inspection [markitdown/_stream_info.py]
              ---> Subclass Dispatch [DocumentConverter.accepts()]
              ---> Concrete Converter [markitdown/converters/*]
              ---> DocumentConverterResult [markdown: str, title: str]

MarkItDown is designed as a dynamic, Pythonic object converter pipeline. Rather than enforcing a single intermediate syntax tree, MarkItDown delegates document parsing to specialized Python libraries or external REST APIs and outputs GFM Markdown directly.

  • Core Python Classes:
    • MarkItDown (markitdown/packages/markitdown/src/markitdown/_markitdown.py): The main registry and orchestration engine maintaining an ordered list of DocumentConverter instances.
    • DocumentConverter (markitdown/packages/markitdown/src/markitdown/_base_converter.py): Abstract base class specifying two primary methods:
      • accepts(file_stream, stream_info, **kwargs) -> bool: Performs stream inspection (MIME type, extension, URL, or magic header sniffing) to determine if the converter can process the payload.
      • convert(file_stream, stream_info, **kwargs) -> DocumentConverterResult: Executes conversion and returns DocumentConverterResult.
    • DocumentConverterResult: Container object encapsulating markdown: str and optional title: Optional[str].
  • Converter Implementations (markitdown/packages/markitdown/src/markitdown/converters/):
    • PdfConverter (_pdf_converter.py): Uses pdfplumber / pypdf for text and layout extraction.
    • DocxConverter (_docx_converter.py): Uses python-docx for Word XML parsing.
    • PptxConverter (_pptx_converter.py): Uses python-pptx for presentation slide extraction.
    • XlsxConverter (_xlsx_converter.py): Uses openpyxl for Excel spreadsheet conversion.
    • HtmlConverter (_html_converter.py): Employs beautifulsoup4 and markdownify for HTML DOM transformation.
    • AudioConverter & _transcribe_audio.py: Speech recognition / OpenAI Whisper integrations.
    • ImageConverter & _llm_caption.py: Multimodal LLM vision integration using openai API clients.
    • DocIntelConverter (_doc_intel_converter.py): Integration with Azure AI Document Intelligence.
    • MarkItDown OCR (markitdown-ocr package): Extends base converters with _pdf_converter_with_ocr.py, _docx_converter_with_ocr.py, and _ocr_service.py (OcrService).

1.3 AnyDoc: High-Speed Pure Rust Document Engine

Input Bytes ---> Content Detection [anydoc/src/formats/detect.rs]
            ---> Format Parser [anydoc/src/formats/*]
            ---> Unified Document Model [anydoc/src/model/mod.rs]
            ---> GFM Serializer [anydoc/src/render/markdown/mod.rs]
            ---> LLM-Ready Markdown Output

AnyDoc is built from the ground up in pure Rust (Edition 2024 / Rust 1.88) to deliver maximum single-pass conversion throughput with single-digit millisecond latency (4.7ms median).

  • Unified Information-Preserving IR (src/model/mod.rs):
    • Document Struct: Self-contained intermediate representation storing blocks: Vec<Block>, notes: Vec<Note>, and embedded assets: Vec<Asset>.
    • Block Enum (src/model/block.rs): Heading { level: u8, anchor: Option<AnchorId>, content: Vec<Inline> }, Paragraph(Vec<Inline>), List(List), Table(Table), BlockQuote(Vec<Block>), CodeBlock { lang: Option<String>, text: String }, Rule.
    • Inline Enum (src/model/inline.rs): Text { text: String, style: Style }, Link { content: Vec<Inline>, target: LinkTarget }, Image { alt: String, source: ImageSource }, Anchor(AnchorId), NoteRef(String), LineBreak.
    • Table Model (src/model/table.rs): High-level table grid model utilizing GridBuilder, CellSlot, and Cell supporting row/col spans, header row identification, and cell alignment.
    • Asset Model (src/model/asset.rs): Retains raw binary byte vectors (Vec<u8>), MIME type, and unique AssetId for embedded media extraction.
  • Native Rust Parsing & Serializing Pipeline:
    • Content Detection (src/formats/detect.rs): Content-based format identification (Format::from_bytes) evaluating magic byte markers (%PDF-, {\rtf, OLE Compound File Binary stream headers via cfb crate, and ZIP package [Content_Types].xml definitions).
    • Format Parsers (src/formats/): Pure Rust modules for docx (using quick-xml), doc (legacy binary Word via cfb), pptx, ppt, xlsx, xls (via calamine), odf (odt/ods/odp), rtf, epub, csv, and pdf (via pdf-inspector).
    • GFM Serializer (src/render/markdown/mod.rs): Unified Markdown renderer enforcing consistent escaping (escape.rs), table formatting (table.rs), and anchor generation (anchors.rs).
    • Polyglot Bindings: Exposes C-FFI, PyO3 Python bindings (python/Cargo.toml), and N-API Node.js bindings (node/Cargo.toml) that release the Python GIL and run on thread pools without event-loop blockage.

2. Comprehensive Format Matrix

2.1 Input and Output Format Support

Format / Capabilities Pandoc (v3.10.1) MarkItDown AnyDoc (v0.1.3)
Primary Language Haskell Python Rust
Input Formats (40+) Markdown, Docx, LaTeX, HTML, EPUB, Org, RST, Typst, DocBook, JATS, Ipynb, ODT, CSV, etc. (No native PDF parser) Docx, Pptx, Xlsx, PDF, HTML, Audio, Images, Zip, Outlook MSG, EPUB, Ipynb, CSV, RSS Docx, Doc, Pptx, Ppt, Xlsx, Xls, Odt, Ods, Odp, Rtf, EPUB, CSV, PDF
Output Formats 50+ (HTML, PDF via pdflatex/typst, Docx, EPUB, LaTeX, Markdown, ICML, Man, RST, etc.) Markdown ONLY (GFM) Markdown & Document IR (with binary assets)
Content Detection File extension & explicit command flags StreamInfo MIME, extension & URL inspection Deep Byte Magic Inspection (%PDF-, {\rtf, OLE CFB, ZIP [Content_Types].xml)
PDF Extraction Requires external tool (pdftotext / pdf2htmlEX) Native via pdfplumber / pypdf Native via pure Rust pdf-inspector crate
Legacy Office (.doc, .ppt, .xls) Unsupported / requires external conversion Partial (.xls via openpyxl/third-party) Native binary parsing (.doc, .ppt, .xls via cfb & calamine)

2.2 Intermediate Representation (IR) Comparison

Dimension Pandoc AST (pandoc-types) MarkItDown Representation AnyDoc IR (Document)
IR Type Fully featured, recursive algebraic data type (Pandoc Meta [Block]) No unified AST; direct string concatenation & library DOMs Strongly typed, information-preserving Rust struct (Document)
Node Mutability Immutable pure functional tree Mutable Python string buffers & DOM trees Mutable Rust struct ownership graph
Embedded Asset Storage In-memory MediaBag (Text.Pandoc.MediaBag) Temporary file system disk dumps First-class binary vectors in Document.assets
Serialization Target Multi-target via custom Writers Single target (Markdown string) Single target (GFM Serializer) or Document struct

2.3 Lossy vs. Lossless Conversion Capabilities

Document Feature Pandoc Capabilities MarkItDown Capabilities AnyDoc Capabilities
Complex Tables (Spans & Align) Lossless AST model (Table colspec & cells); Lossy in plain Markdown writer Lossy (basic HTML table / Markdown pipe conversion) High fidelity (GridBuilder cell slot merging & GFM tables)
Mathematical Formulas Lossless via texmath (LaTeX, MathML, OMML, Typst) Lossy (treated as raw text or omitted) Partial (inline text extraction)
Footnotes & Endnotes Lossless (Note [Block]) Lossy (inline text flattening) Lossless (Note model with Footnote / Endnote kinds)
Nested Lists & Numbering Lossless (retains list start numbers & styles) Partial (markdown list syntax) Lossless (resolves source numbering & nesting)
Embedded Images Extracted to MediaBag or external disk path Generates LLM vision captions or local paths Extracted into Asset byte arrays with AssetId

3. Maximum Capabilities Breakdown

3.1 Extensibility Models

  • Pandoc:
    • Lua Filters: Evaluated via pandoc-lua-engine through hslua. Operates inside the Haskell binary memory space. Highly performant tree traversal using walk / walkM.
    • JSON Filters: Programmatic filters in any language (python-pandocfilters, panflute). Executes as an external subprocess, piping JSON serialized AST over stdin/stdout.
  • MarkItDown:
    • Python Class Inheritance: Developers write custom converters subclassing DocumentConverter in _base_converter.py and register them with MarkItDown(converters=[...]).
    • Plugin Architecture: Modular subpackages like markitdown-ocr and markitdown-sample-plugin.
  • AnyDoc:
    • Native Rust & Language Bindings: Low-level extension via Rust crates, PyO3 Python bindings (python/Cargo.toml), or N-API Node.js bindings (node/Cargo.toml).

3.2 LLM, Multi-Modal & OCR Integrations

MarkItDown Vision/Audio Architecture:
[Image File] ---> ImageConverter ---> _llm_caption.py ---> OpenAI Vision API ---> Image Caption
[Audio File] ---> AudioConverter ---> _transcribe_audio.py ---> Whisper API ---> Audio Transcript
  • Pandoc: Zero built-in LLM/OCR models. Relies entirely on pre-processed inputs or external filter calls.
  • MarkItDown: Built-in multi-modal capabilities:
    • _llm_caption.py invokes OpenAI's Chat Completions API with Vision models (gpt-4o) to generate text descriptions for standalone images or embedded graphics.
    • _transcribe_audio.py uses speech recognition and Whisper to transcribe audio files into Markdown text.
    • markitdown-ocr provides optical character recognition for scanned PDFs and image files using _ocr_service.py (OcrService).
  • AnyDoc: Zero ML runtime overhead in core binary. Delivers ultra-fast, deterministic text extraction (< 5ms) for local pipelines, with scanned page OCR offloaded to hosted services like Firecrawl Parse.

4. Technical Limits & Bottlenecks

4.1 Format Edge Cases & Layout Failures

  1. Pandoc:
    • Cannot natively parse PDF documents without invoking external CLI helpers (pdftotext, pdf2htmlEX).
    • Complex positioning, floating frames, and absolute element layouts in Word (.docx) or CSS HTML are flattened or lost during AST conversion.
  2. MarkItDown:
    • PDF extraction via pdfplumber can misorder multi-column text blocks or overlap header/footer text into main narrative paragraphs.
    • Excel workbook conversion (_xlsx_converter.py via openpyxl) converts calculated formulas into static values and struggles with multi-sheet formatting.
  3. AnyDoc:
    • Scanned PDFs without embedded text layers produce empty outputs unless processed via an external OCR provider.
    • Does not render vector graphics (e.g., DrawingML shapes in PPTX) to SVG; extracts embedded bitmap assets (PNG/JPEG) into Document.assets.

4.2 Memory & Performance Profiles

Benchmarking Throughput (Median Latency per Document):
AnyDoc     : [==] 4.7 ms
Pandoc     : [====================] 102.1 ms
MarkItDown : [========================================] 134.8 ms
LibreOffice: [========================================================================================] 1129.5 ms
  • Pandoc:
    • Median conversion latency: ~102.1 ms.
    • Memory consumption: High heap allocations managed by GHC Garbage Collector when building recursive AST trees for massive documents (>100MB).
  • MarkItDown:
    • Median conversion latency: ~134.8 ms (or several seconds if invoking LLM vision or OCR services).
    • Memory consumption: Heavy Python interpreter and package memory footprints (loading openpyxl, pdfplumber, torch/easyocr). Python GIL restricts multi-threaded scaling.
  • AnyDoc:
    • Median conversion latency: ~4.7 ms (10x-20x faster than Pandoc/MarkItDown).
    • Memory consumption: Extremely lightweight resident set size (< 10MB). LTO-optimized release builds (profile.release: lto = "thin", strip = "symbols") with zero GC pauses.

5. Dependency & System Overhead

5.1 External CLI & Build Complexity

Overhead Metric Pandoc MarkItDown AnyDoc
CLI Runtime Requirements Standalone binary (pandoc-cli); optional LaTeX/Typst/wkhtmltopdf Python 3.10+ interpreter runtime Standalone zero-dependency native binary or npx skill
System Dependencies GHC, Cabal/Stack toolchains (to compile from source) pip, C-extensions for OCR (tesseract, torch) Rust toolchain (cargo, rustc 1.88+)
Sub-process Calls Required for JSON filters & PDF engine invocations Optional for external CLI tool wrappers (exiftool) Zero subprocesses; pure in-process execution
Package Weight Large binary bundle (~50MB - 100MB) Dependent on PyPI packages (~20MB - 500MB+ with PyTorch) Compact static binary (~5MB - 15MB)

6. Decision Framework & Best Practices Matrix

6.1 Enterprise Decision Matrix

Use Case Requirements Recommended Tool Rationale & Architectural Justification
High-Throughput LLM & RAG Ingestion AnyDoc Sub-5ms conversion latency, zero runtime dependencies, robust multi-format support (including legacy .doc/.xls/.ppt), and clean GFM output.
Multi-Target Academic Publishing Pandoc Universal AST (Pandoc Meta [Block]), native citation processing (citeproc), mathematical conversions (texmath), and 50+ output formats.
Multi-Modal AI Ingestion (Vision/Audio) MarkItDown Integrated OpenAI Vision captioning (_llm_caption.py), Whisper audio transcription, and Microsoft Azure Document Intelligence support.
Edge & Serverless Microservices AnyDoc Native Rust runtime with minimal memory footprint (<10MB RSS) and instant cold-start times.
Legacy Scanned Archives (OCR-Heavy) MarkItDown Built-in OCR pipeline via markitdown-ocr package (_ocr_service.py).
Polyglot Monorepos (Node.js / Python) AnyDoc Native PyO3 (python/Cargo.toml) and N-API (node/Cargo.toml) bindings that release the GIL and run asynchronously on libuv thread pools.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment