How to Constrain LLM Outputs with Grammars (GBNF)

Large Language Models are powerful, but they are also unpredictable. When you need structured output — valid JSON, a specific file format, or a constrained natural language response — prompting alone is often unreliable. Grammar-based output constraining solves this problem at the decoding level, guaranteeing that the model's output matches a defined structure. One of the most popular implementations is GBNF (Grammar-Based BNF Format), used by the llama.cpp project and its ecosystem.

What Is GBNF?

GBNF stands for Grammar-Based Backus-Naur Form. It is a format for defining context-free grammars that llama.cpp uses to constrain token generation. Instead of hoping the model produces valid output, the grammar engine masks the logits at each step so that only tokens consistent with the grammar can be sampled. This means the output is guaranteed to be syntactically valid according to your rules.

GBNF is essentially a simplified version of standard BNF with some regex-like conveniences. It supports rules, character ranges, alternations, repetitions, and optional elements. The grammar is compiled into a state machine that tracks which characters are allowed at any given point during generation.

Why Grammar Constraints Matter

Guaranteed validity: JSON, XML, SQL, or custom formats will always parse correctly, eliminating retry loops.
Reduced latency: No need to re-prompt or validate-and-retry when the model produces malformed output.
Lower cost: Fewer wasted tokens on invalid attempts means lower inference costs.
Safety and control: You can restrict the model to a specific vocabulary or structure, preventing hallucinated fields or dangerous commands.
Simpler downstream code: Your parsing logic can be minimal because the structure is guaranteed.

GBNF Syntax Basics

A GBNF grammar is a set of rules. Each rule has a name and a definition. The root rule is named root and is where generation begins. Here is a simple example:

root ::= "Hello, " name "!"
name ::= "World" | "Friend" | "Developer"

This grammar forces the model to output one of three exact strings: Hello, World!, Hello, Friend!, or Hello, Developer!. The | operator represents alternation (choice), and string literals are written in double quotes.

Character ranges use square brackets, similar to regex:

root ::= [a-z]+

This allows one or more lowercase letters. The + suffix means "one or more," * means "zero or more," and ? means "optional."

Constraining JSON Output

The most common use case is forcing valid JSON. Here is a grammar that produces a simple JSON object with a name and age field:

root ::= "{" ws "\"name\"" ws ":" ws string ws "," ws "\"age\"" ws ":" ws number ws "}"

ws ::= [ \t\n]*

string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt])* "\""

number ::= "-"? ([0-9] | [1-9] [0-9]*) ("." [0-9]+)? ([eE] [-+]? [0-9]+)?

This grammar guarantees the output is a JSON object with exactly two keys. The ws rule allows optional whitespace between tokens. The string rule handles escaped characters, and the number rule handles integers, decimals, and scientific notation.

For more flexible JSON, you can allow any valid JSON value:

root ::= value

value ::= object | array | string | number | "true" | "false" | "null"

object ::= "{" ws ( string ws ":" ws value (ws "," ws string ws ":" ws value)* )? ws "}"

array ::= "[" ws (value (ws "," ws value)*)? ws "]"

string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt])* "\""

number ::= "-"? ([0-9] | [1-9] [0-9]*) ("." [0-9]+)? ([eE] [-+]? [0-9]+)?

ws ::= [ \t\n]*

This is the general-purpose JSON grammar shipped with llama.cpp. It allows any valid JSON structure while still guaranteeing parseability.

Using GBNF with llama.cpp

The most direct way to use GBNF is through the llama.cpp CLI. You pass a grammar file using the --grammar-file flag:

./main -m model.gguf -p "Generate a user profile as JSON." --grammar-file json.gbnf -n 256

The model will produce output that strictly conforms to the grammar. Even if the underlying model tends to add markdown code fences or explanatory text, the grammar constraint prevents it.

Using GBNF with Python

For programmatic usage, the llama-cpp-python library provides a clean API. First, install it:

pip install llama-cpp-python

Then use the grammar in your code:

from llama_cpp import Llama

llm = Llama(model_path="model.gguf")

json_grammar = r"""
root ::= "{" ws "\"name\"" ws ":" ws string ws "," ws "\"age\"" ws ":" ws number ws "}"
ws ::= [ \t\n]*
string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt])* "\""
number ::= "-"? ([0-9] | [1-9] [0-9]*) ("." [0-9]+)? ([eE] [-+]? [0-9]+)?
"""

response = llm(
    "Create a fictional character profile.",
    grammar=json_grammar,
    max_tokens=128,
    temperature=0.7,
)

print(response["choices"][0]["text"])

The output will always be valid JSON matching the specified structure, for example:

{"name": "Elara Moonwhisper", "age": 142}

Using GBNF with llamafile

llamafile also supports GBNF grammars. The usage is similar to llama.cpp:

./model.llamafile --grammar-file json.gbnf -p "Describe a planet as JSON."

Building a Custom Grammar for a Specific Schema

Suppose you want the model to generate a list of tasks, each with a priority level restricted to "low," "medium," or "high." Here is a grammar for that:

root ::= "[" ws task (ws "," ws task)* ws "]"

task ::= "{" ws "\"description\"" ws ":" ws string ws "," ws "\"priority\"" ws ":" ws priority ws "}"

priority ::= "\"low\"" | "\"medium\"" | "\"high\""

string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt])* "\""

ws ::= [ \t\n]*

With this grammar, the model can never produce an invalid priority value. The output is guaranteed to be a JSON array of task objects, each with a description string and a priority from the allowed set.

Constraining Natural Language Output

Grammars are not limited to structured data. You can constrain natural language too. For example, to force the model to answer in exactly three sentences:

root ::= sentence " " sentence " " sentence

sentence ::= [A-Z][a-z ]+ "."

This is a simplified example — real English sentences are more complex — but it demonstrates how grammars can enforce structural rules on free text.

Best Practices

Start with the built-in grammars: llama.cpp ships with grammars for JSON, CSV, and other common formats. Use them as a foundation before writing your own.
Keep grammars as permissive as needed: Overly strict grammars can cause the model to produce low-quality output because too many tokens are masked. Allow reasonable flexibility where the model's creativity matters.
Test your grammar independently: Use a GBNF testing tool or write unit tests that verify sample outputs match the grammar before deploying.
Use grammars for structure, prompts for content: The grammar defines the shape; the prompt guides what goes inside. Both work together.
Watch out for performance: Complex grammars with deep nesting can slow down generation because the state machine must evaluate more transitions per token.
Handle whitespace deliberately: Decide whether whitespace is significant in your format and define a ws rule consistently.
Escape special characters in strings: If your grammar includes string literals with special characters, make sure they are properly escaped.
Consider partial JSON parsing: If you stream output, ensure your downstream parser can handle incomplete JSON, since the grammar guarantees validity only when generation completes.

Common Pitfalls

One frequent issue is grammar conflicts where the model gets "stuck" and produces repetitive or empty output. This usually happens when the grammar is too restrictive at a point where the model needs flexibility. For example, forcing exact field values when the model should be generating creative content will degrade quality. The solution is to constrain the structure but leave content generation open.

Another pitfall is forgetting that grammars constrain syntax, not semantics. A grammar can guarantee valid JSON, but it cannot guarantee the JSON contains meaningful or correct data. You still need application-level validation for business logic.

Conclusion

Grammar-based output constraining with GBNF is a powerful technique that bridges the gap between the creative capabilities of LLMs and the reliability requirements of production systems. By defining the allowed structure at the decoding level, you eliminate entire classes of errors — malformed JSON, invalid enum values, unexpected fields — without relying on fragile prompting strategies or retry loops. Whether you are building agents that need structured tool calls, data pipelines that consume model output, or user-facing applications that demand consistent formatting, GBNF gives you deterministic control over the shape of the output while letting the model handle the substance. Start with the built-in JSON grammar, experiment with custom schemas for your specific use case, and combine grammars with thoughtful prompting to get the best of both worlds: creative intelligence and guaranteed structure.