EXPLAINER

Constrained Decoding (Guided Generation)

techniquefreshness: volatilesources not yet verified

Constrained decoding restricts a model's next-token choices at each step so the output must follow a format — a JSON schema, regex or grammar — and cannot be malformed.

Short answer: At every step, the decoder masks out tokens that would break the required format (JSON schema, regex, grammar). The output is guaranteed to be syntactically valid, but valid does not mean correct or well-reasoned.

What is it?

Instead of asking nicely for JSON and hoping, constrained decoding makes it impossible for the model to produce anything else. While generating, the system blocks any next word that would break your format.

That guarantees well-formed output. It does not guarantee the content is true or sensible.

Also known as: guided generation, guided decoding, grammar constrained decoding, schema constrained generation, structured generation, force JSON output, logit masking, token masking.

Technical definition

Implementation compiles a constraint (JSON Schema, regular expression, context-free grammar) into an automaton or pushdown automaton; at each decoding step, the set of tokens that keep the partial output extendable to a valid string is computed and all other logits are masked before sampling. Libraries and engines include Outlines (Willard & Louf, 2023), XGrammar (Dong et al., 2024), llama.cpp GBNF grammars and several serving-engine integrations. Challenges: token boundaries versus grammar symbols, mask-computation overhead, schema features that a given implementation does not support, and possible quality effects when constraints force unnatural tokens.

Entity type: technique · Freshness class: volatile.

When should I use it?

Use when downstream code must parse the output reliably, especially with local models that lack native structured-output features.

Implementation patterns

  • Schema-first: define a JSON Schema, compile it, decode under constraint.
  • Enum/regex constraints for classification labels and IDs.
  • Grammar for small DSLs (SQL subsets, config formats).
  • Constrain the answer but leave a free-text "rationale" field to avoid cramping reasoning.
  • Validate anyway: semantic checks after decoding.

Technologies and frameworks

Commonly associated: Outlines, XGrammar, llama.cpp GBNF, vLLM guided decoding backends, provider structured-output features. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Decide the target format and write the schema.
  2. Check what your engine/provider supports.
  3. Include "unknown"/null options.
  4. Decode under constraint and validate semantics.
  5. Test with adversarial inputs (How to Evaluate AI Systems and Outputs).

Common failure modes

  • Valid but wrong content: a perfectly formed field filled with a hallucinated value.
  • Unsupported schema keywords silently ignored or rejected by the implementation.
  • Truncation by max-tokens leaving incomplete JSON even under constraints.
  • Over-constraining so the model cannot express uncertainty or "unknown".
  • Latency overhead from grammar compilation on large schemas.

Troubleshooting

  • Field values wrong though JSON valid — Add enums/nullable "unknown", validate semantically, ground with retrieval.
  • Schema rejected — Check which JSON Schema features your engine or provider supports; simplify.
  • Incomplete JSON — Raise max tokens; keep schemas small.

Contrasted with

Questions this page answers

  • What is constrained decoding?
  • How do I force an LLM to output valid JSON?
  • How does guided generation work?
  • Does constrained decoding reduce quality?
  • What is the difference between JSON mode and constrained decoding?

Typical goals: guarantee valid JSON from an LLM; choose between prompting and constraints for structure; use grammars with local models.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Which engines and providers support which schema features.
  • Performance and quality claims for specific libraries.

Canonical sources (unverified leads):

  • Willard & Louf, Efficient Guided Generation for Large Language Models (2023) — arXiv:2307.09702
  • Dong et al., XGrammar: Flexible and Efficient Structured Generation Engine for LLMs (2024) — arXiv:2411.15100
  • Geng et al., Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (2023) — arXiv:2305.13971
Learning path: Structured output and constrained decoding (step 4 of 6)
← JSON SchemaGrammar-Guided Generation (CFG and Regex Constraints) →