Generating Valid JSON from LLMs: A Guide to Outlines

Introduction to Outlines

Large Language Models (LLMs) are powerful text generators, but getting them to produce structured output reliably has long been a pain point. If you've ever asked an LLM to return JSON and received malformed syntax, missing keys, or hallucinated fields, you're not alone. Outlines is an open-source Python library that solves this problem by constraining LLM generation at the token level, guaranteeing that the output matches a specified schema.

Unlike prompt-based approaches that merely ask the model to "please return valid JSON," Outlines modifies the generation process itself. It works by manipulating the logits of the next-token distribution so that only tokens consistent with the desired format can be sampled. The result is guaranteed valid output, every single time.

Why Structured Output Matters

Structured output is essential whenever an LLM's response needs to be consumed programmatically. Common scenarios include:

• API integrations where the LLM output feeds into downstream services expecting specific data shapes.
• Data extraction tasks like pulling entities, dates, or relationships from unstructured text.
• Multi-step pipelines where one LLM call's output becomes another's input.
• Function calling where arguments must conform to a function signature.
• Agent systems that parse tool calls from model responses.

Without guarantees, developers resort to brittle workarounds: retry loops, regex post-processing, fallback prompts, and defensive parsing. These add latency, cost, and complexity. Outlines eliminates this entire class of problems.

How Outlines Works

Outlines operates by compiling a target schema—whether a regex, JSON schema, Pydantic model, or grammar—into a finite state machine (FSM). At each generation step, the FSM determines which tokens are valid given the text produced so far. Outlines then masks the logits of invalid tokens to negative infinity before sampling, making it impossible for the model to produce an invalid token.

This approach has several important properties:

• It is model-agnostic within supported backends, working with Transformers, vLLM, llama.cpp, and others.
• It introduces negligible overhead because the FSM is compiled once and reused.
• It does not require fine-tuning or special prompt engineering.
• It preserves the model's capabilities—the model still chooses among valid tokens based on its learned distribution.

Installation

Outlines is available on PyPI and can be installed with pip. The core package supports multiple backends, so you may also want to install your preferred inference engine.

pip install outlines
# For the Transformers backend
pip install transformers torch
# For the vLLM backend (high-throughput serving)
pip install vllm

Outlines supports several backends out of the box, including Hugging Face Transformers, vLLM, llama.cpp, Mamba, and OpenAI-compatible APIs. This tutorial focuses on the local Transformers backend for clarity, but the API is consistent across backends.

Basic Usage: Generating JSON

The most common use case is generating JSON that conforms to a schema. Outlines provides a clean, decorator-based API. Let's start with a simple example using a Pydantic model.

import outlines
from pydantic import BaseModel

# Define the schema you want the model to produce
class Person(BaseModel):
    name: str
    age: int
    occupation: str

# Load a model (Outlines wraps it for constrained generation)
model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

# Create a generator function bound to the schema
@outlines.generate.json(model, Person)
def generate_person(prompt: str):
    return prompt

# Call it like a normal function
person = generate_person("Generate a profile for a fictional software engineer.")
print(person)
# Output: name='Elena Vasquez' age=29 occupation='Software Engineer'

Notice that the return value is not a string—it's an actual Person instance. Outlines parses the constrained output into the Pydantic model automatically. This means you get type safety and validation for free.

Using Raw JSON Schema

If you don't use Pydantic, or if you're working with externally defined schemas, you can pass a JSON Schema dictionary directly.

import outlines
import json

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "completed": {"type": "boolean"},
        "priority": {"type": "integer", "minimum": 1, "maximum": 5}
    },
    "required": ["title", "completed", "priority"]
}

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.json(model, schema)
def generate_task(prompt: str):
    return prompt

task = generate_task("Create a task for finishing the quarterly report.")
print(json.dumps(task, indent=2))

The output will always be valid JSON matching the schema, including respecting constraints like minimum and maximum.

Generating Other Structured Formats

Regex-Constrained Text

Sometimes you need text that matches a pattern but isn't JSON—for example, phone numbers, dates, or identifiers. Outlines supports regex constraints.

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.regex(model, r"\d{3}-\d{3}-\d{4}")
def generate_phone(prompt: str):
    return prompt

phone = generate_phone("Give me a phone number for a New York office.")
print(phone)  # e.g., "212-555-0147"

Choice and Type Constraints

For classification tasks where the output must be one of several options, use the choice generator.

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.choice(model, ["positive", "negative", "neutral"])
def classify_sentiment(prompt: str):
    return prompt

sentiment = classify_sentiment("Classify: 'The food was amazing but service was slow.'")
print(sentiment)  # "mixed" is impossible—only the three choices are valid

Similarly, generate.format constrains output to a Python type like int, float, bool, or str.

@outlines.generate.format(model, int)
def generate_number(prompt: str):
    return prompt

result = generate_number("What is 47 multiplied by 3?")
print(result)  # 141 (as an integer, not a string)

Grammar-Based Generation

For more complex structured languages, Outlines supports context-free grammars (CFGs) via the Lark library. This is useful for generating SQL queries, code snippets, or custom DSLs.

import outlines

# A simple arithmetic grammar
arithmetic_grammar = """
start: expr
expr: term (("+" | "-") term)*
term: factor (("*" | "/") factor)*
factor: NUMBER | "(" expr ")"
%import common.NUMBER
%import common.WS
%ignore WS
"""

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.cfg(model, arithmetic_grammar)
def generate_expression(prompt: str):
    return prompt

expr = generate_expression("Write an arithmetic expression for compound interest.")
print(expr)  # e.g., "(1000 * (1 + 5)) * 12"

Working with Chat Models

Many modern models use a chat format with system, user, and assistant messages. Outlines handles this through the outlines.generate functions combined with chat-template-aware models. For instruction-tuned models, you should format your prompt appropriately.

import outlines
from pydantic import BaseModel

class Summary(BaseModel):
    main_point: str
    key_facts: list[str]
    sentiment: str

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.json(model, Summary)
def summarize(prompt: str):
    return prompt

text = """
The quarterly earnings report showed a 15% increase in revenue,
driven primarily by strong sales in the European market. However,
supply chain disruptions led to increased costs, slightly reducing
overall profit margins compared to the previous quarter.
"""

result = summarize(f"Summarize the following text:\n\n{text}")
print(result.main_point)
print(result.key_facts)
print(result.sentiment)

Using the vLLM Backend for Production

For production workloads with high throughput requirements, the vLLM backend is recommended. It supports batched inference and paged attention, making it significantly faster for concurrent requests.

import outlines
from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool
    tags: list[str]

# Use vLLM backend instead of Transformers
model = outlines.models.vllm(
    "mistralai/Mistral-7B-Instruct-v0.2",
    tensor_parallel_size=1
)

@outlines.generate.json(model, Product)
def generate_product(prompt: str):
    return prompt

product = generate_product("Generate a product listing for a wireless keyboard.")
print(product)

Best Practices

Choose the Right Schema Granularity

Overly complex schemas can slow down generation and confuse smaller models. Keep schemas as simple as possible while still capturing the structure you need. Use nested objects sparingly, and prefer flat structures when feasible.

Provide Clear Prompts

Outlines guarantees valid structure, but it does not guarantee semantically correct content. A clear, specific prompt still matters. Tell the model exactly what fields mean and what kind of values you expect.

# Vague prompt—may produce technically valid but useless JSON
prompt = "Generate a person."

# Better prompt—guides the model toward useful output
prompt = """Generate a realistic profile for a fictional character
who is a marine biologist living in coastal Maine.
The name should be plausible for the region.
The age should be between 30 and 60.
The occupation should relate to marine biology."""

Use Enumerations for Constrained Fields

When a field can only take specific values, use enums or JSON Schema enums. This prevents the model from generating unexpected values.

from pydantic import BaseModel
from enum import Enum

class Priority(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"

class Task(BaseModel):
    description: str
    priority: Priority
    estimated_hours: float

import outlines
model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")

@outlines.generate.json(model, Task)
def generate_task(prompt: str):
    return prompt

task = generate_task("Create a task for reviewing pull requests.")
print(task.priority)  # Always one of: low, medium, high

Handle Optional Fields Explicitly

Pydantic's Optional types and JSON Schema's nullable properties are supported, but be deliberate about which fields are required versus optional. Too many optional fields can lead to sparse, unhelpful output.

Cache the Compiled FSM

Outlines compiles the schema into an FSM the first time a generator is called. If you're serving many requests with the same schema, reuse the generator function rather than recreating it. The FSM is cached automatically within a single process.

Test with Smaller Models First

Constrained generation works with any model, but smaller models may produce lower-quality content within the constraints. Test your pipeline with a small model during development, then scale up to a larger model for production quality.

Common Pitfalls

• Expecting semantic correctness from structure alone: Outlines guarantees format, not accuracy. A model can produce valid JSON with factually wrong values.
• Using overly strict schemas: If your schema is too rigid, the model may struggle to fill in all fields meaningfully, leading to repetitive or generic values.
• Ignoring token limits: Complex schemas with many fields may require more tokens. Ensure your max_tokens setting is high enough to complete the structured output.
• Forgetting to handle the parsed result: When using Pydantic schemas, the return value is a model instance, not a string. Don't try to json.loads() it again.

Conclusion

Outlines represents a fundamental shift in how developers work with LLMs for structured output tasks. By constraining generation at the token level, it eliminates an entire category of reliability problems that have plagued LLM applications since their inception. Whether you're building data extraction pipelines, agent systems, or API integrations, Outlines lets you focus on the semantics of your application rather than wrestling with malformed JSON and retry logic. By combining clear prompts, well-designed schemas, and an appropriate inference backend, you can build production-grade systems that produce guaranteed-valid structured output on every single call. As the ecosystem around structured generation matures, tools like Outlines are becoming an essential part of the modern LLM developer's toolkit.