PR: #553 (consolidated from earlier stacked PRs #538-#541)
I've been using qmd to index agent instruction files and Obsidian-style notes and noticed chunks landing in weird places. Pulled the thread and it turned into four distinct improvements, shipped as one PR with clean commit-per-change history.
The chunker uses scored "break points" (places where splitting is OK) and "protected regions" (places to never split). Four specific issues:
1. Code fence detection pairs by toggle, not by length. findCodeFences matches /\n```/g and flips a boolean on every match. A fence opened with four backticks never gets recognized. A stray triple-backtick inside a longer fence prematurely closes it. Tilde fences don't exist at all. These aren't edge cases. Agent prompt files routinely wrap three-backtick examples in four-backtick fences, and tutorials do the same. Today the chunker splits right through them.
2. List detection is a two-line regex with score 5. /\n[-*]\s/g and /\n\d+\.\s/g, both scored 5, so low that any nearby heading (50-100), blank line (20), or code fence boundary (80) always wins. On long lists, the chunker splits between paragraphs inside items instead of between items. Sublists aren't detected at all. 1) numbering isn't detected at all. End-of-list transitions (arguably the single most valuable break point in a document that contains lists) get no special treatment.
3. No understanding of structural XML tags. Agent docs wrap instructions in <example>, <instructions>, <thinking>, <tool_use>, <system>. These mark coherent blocks that should stay together. The chunker treats them as opaque text and splits them wherever whitespace happens to fall.
4. The CodeFenceRegion type is code-fence-specific. Contract issue, not behavior. But it blocks adding new producers of protected regions without a rename.
Five commits, each self-contained:
1. Fence pairing fix. Rewrites findCodeFences to track the opening fence's character and length. Close candidates must use the same char, be at least as long, and carry no info string. Same-length non-nesting is deterministically resolved per CommonMark spec: you can't nest a 4-backtick inside a 4-backtick, you need 5 outside. 12 new tests.
2. Rename. CodeFenceRegion becomes ProtectedRegion with an optional kind tag. No behavior change. Kept as its own commit so the rename noise is isolated from the feature work.
3. List-aware scanner. Replaces the naive regex patterns with a stack-based scanner that tracks nested list frames. Depth-weighted scoring:
- Top-level item: 70 (same as h4)
- Second-level: 45
- Third-level and deeper: 25
- List-end transition: 75. Third-highest in the whole score table.
16 new tests. Picks up 1) ordered markers and proper sublist detection as freebies.
4. XML tag break points. Detects line-anchored paired tags and emits asymmetric break points:
tag-open: 30 (low; splitting right before content orphans the opener)tag-close: 75 (same score as list-end; closing a structured block is a great place to split)
HTML5 element names are rejected via a hard-coded blocklist so inline HTML in markdown doesn't get picked up as structural. Tags inside code fences are ignored. 27 new tests.
5. Test isolation fix. Mirrors upstream commit 66e70c0. The createStore throws without explicit path in test mode test was flaky on Bun ubuntu due to _productionMode state leaking between test files.
A lot of what I index in qmd is Obsidian-style: lists of thoughts, bullet-point outlines, nested meeting notes, sublists under project headings. Today those get sliced between paragraphs inside items, losing the structure a reader relies on to make sense of them. After this change, long lists split at item boundaries, nested sublists stay coherent with their parents, and end-of-list transitions become strong signals. The effect on retrieval is noticeable on my own corpus.
The XML tag work is narrower. It matters specifically for agent instruction files and prompt collections, which are a real-but-niche use case. The interesting thing is that the same scoring principle ("high score on the end of a structured block, low on the start") applies to both lists and tags.
Every change includes documented scope cuts. The short version: I optimized for "don't degrade anything that works today" rather than "be spec-compliant with CommonMark." A few examples:
- Same-indent marker transitions. CommonMark says
- foofollowed by* barat the same indent ends one list and starts another. I treat them as one list. For chunking, "same indent = same list" produces visibly better splits. - Lazy continuation, loose/tight list distinction, 4-space indented code blocks. Deferred. Rendering concerns, not chunking concerns.
- Mid-line XML tags.
Here's an <example>foo</example>is ignored. Inline tag usage isn't a structural signal. - Column-0 fences only. Indented fences are not detected. A future list-aware pass that understands indentation context could handle this.
Each of these is documented in a block comment next to the code.
The changes sit on clean seams. The commits are ordered so any prefix is self-contained, and the two feature commits (list-aware and XML tags) are independent of each other. Happy to split into separate PRs if that's preferable for review or landing incrementally.
An opinion on whether this is a direction you want the chunker to go. The fence fix is the obvious "yes" if anything is. The list and XML scanners are where I'd expect scrutiny. The design decisions are documented in the PR description and I'd rather discuss them than defend them.
If any piece is misaligned with how you think about the chunker, happy to adjust, reshape, or drop it. None of this was solicited and I'm not precious about it.