Design: SentencePiece post-processing (PR #7625)

design.md

Design: Post-processing (special-token templates) for Microsoft.ML.Tokenizers

Status: prototype (PR #7625). Shareable design asset for review.

Problem

SentencePieceTokenizer.CreateFromTokenizerJson loads Hugging Face JSON-only Unigram tokenizers. HF tokenizer.json declares how special tokens wrap an encoded sequence in a post_processor block. The library has no post-processing abstraction today; SentencePiece only supports a single BOS + single EOS via the addBeginningOfSentence / addEndOfSentence switches. That cannot represent common real-world templates.

Data (sampled popular Unigram models on HF, n=57 resolved)

Shape	Count	Families
Representable (<=1 prefix + <=1 suffix)	37	xlm-roberta, albert, t5/flan-t5, camembert, deberta-v3, pegasus, bigbird, fnet
NOT representable (multi-token)	14	XLNet, mBART, NLLB with multi-token suffixes
null (no post-processor)	5	reformer
Different processor type (RobertaProcessing)	1	camembert variant

~25% need more than single-BOS/single-EOS. Conclusion: worth supporting now.

Design (prototype scope)

Generalize SentencePiece special-token wrapping from a single BOS/EOS to ordered prefix and suffix special-token lists:

• addBeginningOfSentence gates the whole prefix list.
• addEndOfSentence gates the whole suffix list.
• Common case prefix=[], suffix=[] is byte-for-byte the current behavior.
• Lists carry (int Id, string Token); applied at every encode / count / index site.

Parser (post_processor block)

• TemplateProcessing: Scan single template; SpecialToken before Sequence => prefix; after => suffix. Resolve ids via special_tokens -> added_tokens -> vocab.
• RobertaProcessing: [] prefix, [] suffix.
• BertProcessing: [[CLS]] prefix, [[SEP]] suffix.
• Sequence: Recursively process child processors.
• Interleaved/mid-sequence specials => NotSupportedException.

added_tokens (special==true) auto-populate special-token map and are classified Control. model.byte_fallback is read instead of hardcoded false.

Out of scope for the prototype

• General Tokenizer-level PostProcessor abstraction (reusable for BPE/WordPiece/Tiktoken) + richer Encoding.
• Pair-sequence encoding + type_ids.
• Dynamic language tokens for mBART/NLLB.
• ByteLevel post-processor trim_offsets.

Open questions for reviewers

1. Should the public abstraction live at the Tokenizer base now, or stay SentencePiece-internal until a second consumer needs it?
2. Do we want a public way to inspect/override the resolved prefix/suffix lists?
3. Behavior for unrepresentable templates: throw (current choice) vs. best-effort?

Performance Results

Perf testing on 1000 iterations (hot paths):

• EncodeToIds: single-affix baseline 6.33 ms -> multi-affix 4.49 ms (-29% faster)
• CountTokens: single-affix baseline 7.05 ms -> multi-affix 4.98 ms (-29% faster)
• EncodeToTokens: single-affix baseline 13.56 ms -> multi-affix 10.52 ms (-22% faster)

? Full test suite: 393/393 pass (net8.0 and net48) ? No regression detected ? actually 20-30% improvement on common case.