Native support of grammar-based Decoding in inference servers

native-grammar-decoding.md

Native support of grammar-based Decoding in inference servers

This note distinguishes server-side constrained decoding from prompt-only formatting. In constrained decoding, the inference server masks invalid next tokens while generating; it does not merely ask the model to follow a format and parse the finished text.

Constraint interfaces

Interface	Expressive power	Typical use
JSON Schema	Structured JSON objects and arrays	API responses, function arguments
Regex	Regular languages	IDs, dates, fixed records
Choice / enum	One of finitely many strings	Classification, routing
Context-free grammar	General recursive syntax	SQL, patches, XML, DSLs
Structural tags	Fixed tag wrappers with a constrained payload	XML-like tool-call envelopes

JSON Schema is often implemented by compiling it to a grammar internally. Grammar syntax is not universal: Lark, EBNF, and GBNF are distinct notations for closely related ideas.

OpenAI

OpenAI custom tools accept a Lark grammar for the tool input. This is an endpoint-level grammar-constrained transport, separate from classic JSON-schema function calling.

• Custom tools guide

vLLM

vLLM's OpenAI-compatible server supports decoder-time structured outputs for:

• choice
• regex
• json (JSON Schema)
• grammar (context-free EBNF)
• structural_tag

structural_tag is the closest first-class XML-like facility. A request declares fixed begin and end strings plus a JSON Schema for the material between them. For example, the server can constrain:

<function=get_weather>{"city":"Berlin"}</function>

Once generation reaches the trigger prefix, the opening tag, JSON payload, and closing tag are constrained by the structured-output backend. This is still not a complete tool API: the application must advertise the function in the prompt, parse the completed content, and dispatch it.

• Structured-output formats
• Structural-tag request example
• XGrammar backend compilation

llama.cpp

llama-server supports JSON Schema and a grammar request field using GBNF (GGML BNF). GBNF can constrain arbitrary formal output languages, including a full XML tool-call dialect. JSON Schema can be translated to GBNF by the server.

With the optional LLGuidance integration enabled, llama.cpp also accepts a Lark-like grammar path.

• llama.cpp server schema
• GBNF guide
• LLGuidance integration

Tool calling versus constrained text

Native tool APIs generally use structured JSON events: OpenAI tool_calls, Anthropic tool_use, and Gemini function-call parts. They have tool identity, call IDs, schemas, and tool-result continuation as part of the API protocol.

A grammar- or structural-tag-constrained response guarantees syntax, but normally remains ordinary output content. It needs a client-side parser and dispatcher. That can be an excellent design for a local agent, but it should not be represented as native provider tool transport.

Suggested probes

A capability probe should test each transport independently:

1. JSON Schema: request a strict schema and validate the returned value.
2. Lark / EBNF / GBNF grammar: request a non-JSON language such as an apply_patch envelope and parse it with the supplied grammar.
3. vLLM structural tags: request <function=name>{...}</function> with a JSON Schema payload; validate wrapper, payload, and rejection behavior for malformed requests.
4. Native tools: separately verify that the endpoint returns structured tool events, and that tool-result continuation works.

This separation avoids conflating prompt-following XML, constrained text, and native tool invocation.