What is it?
Sometimes the output is not JSON but some other strict format: a date pattern, a tiny query language, a command. A grammar describes exactly which strings are legal, and the decoder uses it to rule out illegal continuations as the model writes.
Also known as: CFG decoding, context free grammar LLM, regex constrained generation, grammar sampling, formal grammar decoding, grammar based sampling.
Technical definition
Regular expressions map to finite-state machines; context-free grammars require pushdown-style parsing (e.g., Earley/LR-style or specialised engines). The decoder intersects the grammar state with the tokenizer vocabulary to compute allowed tokens per step. Pitfalls: ambiguity between grammar terminals and sub-word tokens, left recursion, and large vocabularies slowing mask construction. See GBNF Grammars (llama.cpp) for llama.cpp's grammar format and Constrained Decoding (Guided Generation) for the general technique.
Entity type: technique · Freshness class: evolving.
When should I use it?
Use for small, strict formal outputs, especially with local engines; for ordinary JSON, a schema is simpler.
Implementation patterns
- Regex for identifiers, dates and enums.
- Small CFG for DSLs and command languages.
- Grammar + post-validation (execute in a sandbox or parse with a real parser).
- Keep grammar minimal and extend only when tests fail.
Technologies and frameworks
Commonly associated: llama.cpp GBNF, Outlines, XGrammar, Lark-style grammar tools. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define the language with examples of valid/invalid strings.
- Write the minimal grammar.
- Test the grammar with a parser, independent of the model.
- Decode under the grammar.
- Validate semantics and run in a sandbox.
Common failure modes
- Grammar valid but semantically harmful (valid SQL that deletes rows).
- Tokenizer mismatch producing dead ends.
- Over-large grammars making decoding slow.
- Left-recursive or ambiguous rules not supported by the engine.
Troubleshooting
- Generation stalls or ends early — Check the grammar for dead ends and ensure an end-of-sequence path is allowed.
- Slow decoding — Simplify the grammar; precompile; use engine-recommended features.
- Valid but unsafe commands — Whitelist operations in the grammar and run in a sandbox with least privilege.
Contrasted with
- JSON Schema — JSON Schema describes JSON documents only; a grammar can describe any formal language.
Questions this page answers
- How do I use a grammar to constrain an LLM?
- Can LLMs generate output that matches a regex?
- What is grammar-based sampling?
- How do I generate valid SQL with an LLM?
Typical goals: constrain output to a small formal language; write a grammar for an LLM.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Geng et al., Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (2023) — arXiv:2305.13971
- Scholak et al., PICARD (2021) — arXiv:2109.05093
- llama.cpp grammar documentation — github.com/ggml-org/llama.cpp (locate grammars README; repository ownership has changed historically)