Introduction to vLLM for Data Extraction
Data extraction is one of the most valuable applications of large language models in production environments. Whether you are parsing invoices, extracting structured information from legal documents, or transforming unstructured text into database-ready records, the ability to reliably convert free text into structured data is a foundational capability for modern applications. vLLM, an open-source inference engine, has emerged as a powerful tool for this workflow because it delivers high-throughput, low-latency inference while maintaining compatibility with the OpenAI API format.
This tutorial walks through building a complete data extraction pipeline using vLLM. We will cover what vLLM is, why it is well-suited for extraction tasks, how to set up a production-grade pipeline, and the best practices that separate a fragile prototype from a reliable system.
What is vLLM?
vLLM is a high-throughput inference engine designed specifically for large language models. Developed originally at UC Berkeley, it introduced PagedAttention, a technique that manages the KV cache memory in a way inspired by operating system virtual memory paging. This allows vLLM to serve multiple concurrent requests efficiently, dramatically improving GPU utilization compared to naive implementations.
For data extraction workloads, vLLM offers several distinct advantages:
- High throughput: PagedAttention and continuous batching allow vLLM to process many extraction requests in parallel, which matters when you have thousands of documents to process.
- OpenAI-compatible API: vLLM exposes a server that mirrors the OpenAI Chat Completions API, so existing client code and libraries like the official OpenAI Python SDK work with minimal changes.
- Local deployment: You control the model, the hardware, and the data. Sensitive documents never leave your infrastructure, which is critical for compliance-heavy industries.
- Quantization support: vLLM supports AWQ, GPTQ, and FP8 quantization, letting you run capable models on more modest hardware.
- Structured output: Through guided decoding integrations with libraries like Outlines and LM-Format-Enforcer, vLLM can constrain model output to valid JSON schemas, which is essential for reliable extraction.
Why vLLM Matters for Data Extraction
Data extraction pipelines have a specific shape that makes them demanding. You typically have a large batch of heterogeneous documents, each requiring the model to produce a structured, machine-readable output. The failure modes are well known: models hallucinate fields, produce malformed JSON, omit required keys, or invent values that do not appear in the source text. A naive approach of calling an API per document and hoping the output parses correctly will not survive contact with production data.
vLLM addresses these challenges on two fronts. First, its throughput characteristics mean you can process large document volumes economically. A single GPU running vLLM can often replace dozens of parallel API calls to a hosted endpoint. Second, its support for guided decoding lets you force the model to emit output that conforms to a JSON schema, eliminating an entire class of parsing failures before they happen.
The combination of speed and determinism is what makes vLLM compelling. You get the flexibility of a general-purpose language model with the reliability of a purpose-built parser.
Setting Up the Environment
Before building the pipeline, you need a working vLLM installation. The recommended approach is to use a Python virtual environment on a machine with an NVIDIA GPU. vLLM requires CUDA 12.1 or higher and a GPU with compute capability 7.0 or above.
# Create and activate a virtual environment
python -m venv vllm-env
source vllm-env/bin/activate
# Install vLLM
pip install vllm
# Install supporting libraries
pip install openai pydantic outlines
Once installed, you can launch a vLLM server. For extraction tasks, a model like Qwen2.5-7B-Instruct or Llama-3.1-8B-Instruct offers a strong balance of capability and resource requirements. The following command starts a server with guided JSON decoding enabled:
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8000 \
--max-model-len 8192 \
--guided-decoding-backend outlines \
--gpu-memory-utilization 0.9
The --guided-decoding-backend outlines flag is important. It tells vLLM to use the Outlines library for structured output generation, which we will rely on to enforce JSON schemas during extraction.
Defining the Extraction Schema
The foundation of any extraction pipeline is a clear schema that describes what you want to extract. Pydantic is the natural choice here because it provides validation, type checking, and automatic JSON schema generation. Let us define a schema for extracting information from invoices, a common extraction use case.
from pydantic import BaseModel, Field
from typing import Optional, List
from datetime import date
class InvoiceLineItem(BaseModel):
description: str = Field(..., description="Description of the product or service")
quantity: float = Field(..., description="Number of units")
unit_price: float = Field(..., description="Price per unit in the invoice currency")
total: float = Field(..., description="Total price for this line item")
class Invoice(BaseModel):
invoice_number: str = Field(..., description="Unique invoice identifier")
invoice_date: str = Field(..., description="Date the invoice was issued")
due_date: Optional[str] = Field(None, description="Date payment is due")
vendor_name: str = Field(..., description="Name of the company issuing the invoice")
vendor_address: Optional[str] = Field(None, description="Address of the vendor")
customer_name: str = Field(..., description="Name of the customer or client")
customer_address: Optional[str] = Field(None, description="Address of the customer")
line_items: List[InvoiceLineItem] = Field(..., description="List of billed items")
subtotal: float = Field(..., description="Sum before tax")
tax_amount: float = Field(0.0, description="Total tax amount")
total_amount: float = Field(..., description="Final total including tax")
currency: str = Field("USD", description="Three-letter currency code")
Notice the use of Field with descriptions. These descriptions serve double duty: they document the schema for developers and they are included in the JSON schema that gets passed to the model, helping the model understand exactly what each field should contain.
Building the Extraction Client
With the schema defined and the vLLM server running, we can build the extraction client. Because vLLM exposes an OpenAI-compatible API, we use the standard OpenAI Python SDK pointed at our local server.
import json
from openai import OpenAI
from pydantic import BaseModel, ValidationError
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed-for-local-vllm",
)
def build_extraction_prompt(document_text: str, schema: type[BaseModel]) -> list[dict]:
"""Build a chat prompt for structured extraction."""
schema_json = json.dumps(schema.model_json_schema(), indent=2)
return [
{
"role": "system",
"content": (
"You are a precise data extraction assistant. "
"Extract structured information from the provided document. "
"Only use information that is explicitly present in the text. "
"If a field is not present, use null for optional fields. "
"Do not guess or hallucinate values. "
"Return valid JSON matching the provided schema."
),
},
{
"role": "user",
"content": (
f"Extract invoice data from the following document.\n\n"
f"Target JSON schema:\n{schema_json}\n\n"
f"Document:\n{document_text}"
),
},
]
def extract_document(
document_text: str,
schema: type[BaseModel],
model: str = "Qwen/Qwen2.5-7B-Instruct",
) -> BaseModel:
"""Extract structured data from a document using vLLM."""
messages = build_extraction_prompt(document_text, schema)
schema_dict = schema.model_json_schema()
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=0.0,
max_tokens=2048,
extra_body={
"guided_json": schema_dict,
},
)
raw_output = response.choices[0].message.content
try:
parsed = json.loads(raw_output)
return schema.model_validate(parsed)
except (json.JSONDecodeError, ValidationError) as e:
raise ValueError(f"Extraction failed: {e}\nRaw output:\n{raw_output}")
The critical piece here is the extra_body parameter containing guided_json. This is vLLM-specific and instructs the server to constrain the model's output to the provided JSON schema during generation. With guided decoding enabled, the model physically cannot produce invalid JSON or omit required fields, because the decoder masks tokens that would violate the schema at every step.
Processing Documents in Batches
Individual extraction calls are useful for prototyping, but real pipelines process documents in bulk. vLLM's continuous batching shines here. Rather than sending requests one at a time and waiting, you can submit many requests concurrently and let vLLM batch them internally for maximum GPU utilization.
import concurrent.futures
from typing import List, Tuple
def extract_batch(
documents: List[str],
schema: type[BaseModel],
model: str = "Qwen/Qwen2.5-7B-Instruct",
max_workers: int = 32,
) -> List[Tuple[int, BaseModel | None, str | None]]:
"""
Extract structured data from a batch of documents concurrently.
Returns a list of (index, result, error) tuples.
"""
results: List[Tuple[int, BaseModel | None, str | None]] = [None] * len(documents)
def _extract_one(index: int, text: str):
try:
result = extract_document(text, schema, model)
return (index, result, None)
except Exception as e:
return (index, None, str(e))
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [
executor.submit(_extract_one, i, doc)
for i, doc in enumerate(documents)
]
for future in concurrent.futures.as_completed(futures):
index, result, error = future.result()
results[index] = (index, result, error)
return results
# Example usage
if __name__ == "__main__":
sample_documents = [
"""
INVOICE #INV-2024-0042
Date: March 15, 2024
Due: April 14, 2024
From: Acme Supplies Inc.
123 Industrial Way, Springfield, IL 62701
Bill To: Globex Corporation
456 Corporate Blvd, Austin, TX 78701
Items:
- Widget A, 100 units @ $2.50 = $250.00
- Widget B, 50 units @ $5.00 = $250.00
- Widget C, 25 units @ $10.00 = $250.00
Subtotal: $750.00
Tax (8%): $60.00
Total: $810.00
""",
"""
INVOICE
Invoice Number: 99283-A
Issued: 2024-06-01
Vendor: TechParts LLC
Customer: DataCorp
Line Items:
1. Cable HDMI 2m, qty 200, $4.50 each, total $900.00
2. USB-C Adapter, qty 75, $12.00 each, total $900.00
Subtotal: $1800.00
Tax: $144.00
Total Due: $1944.00
Currency: USD
""",
]
batch_results = extract_batch(sample_documents, Invoice)
for index, result, error in batch_results:
if error:
print(f"Document {index} failed: {error}")
else:
print(f"Document {index}: {result.vendor_name} - {result.total_amount} {result.currency}")
Using a thread pool with a high worker count works well because each call to the vLLM server is an I/O-bound HTTP request. The server receives many near-simultaneous requests and batches them onto the GPU. You should tune max_workers based on your hardware; a good starting point is 16 to 64 concurrent requests for a single GPU.
Handling Edge Cases and Validation
Even with guided decoding, you should treat model output as untrusted data. Guided decoding guarantees syntactically valid JSON that matches the schema, but it does not guarantee semantic correctness. The model might still extract a wrong value, misread a number, or populate an optional field with null when a value was actually present.
A robust pipeline includes a validation layer that checks for common issues:
from pydantic import BaseModel, field_validator
class ValidatedInvoice(Invoice):
@field_validator("total_amount")
@classmethod
def check_total(cls, v, info):
values = info.data
subtotal = values.get("subtotal", 0)
tax = values.get("tax_amount", 0)
expected = round(subtotal + tax, 2)
if abs(v - expected) > 0.01:
raise ValueError(
f"total_amount {v} does not match subtotal + tax ({expected})"
)
return v
@field_validator("line_items")
@classmethod
def check_line_items(cls, v):
for item in v:
expected = round(item.quantity * item.unit_price, 2)
if abs(item.total - expected) > 0.01:
raise ValueError(
f"Line item total {item.total} does not match "
f"qty * price ({expected})"
)
return v
def extract_with_validation(document_text: str) -> ValidatedInvoice:
"""Extract and validate, retrying once on validation failure."""
try:
return extract_document(document_text, ValidatedInvoice)
except ValueError:
# Retry with a more explicit prompt
messages = build_extraction_prompt(document_text, ValidatedInvoice)
messages.append({
"role": "user",
"content": (
"The previous extraction had validation errors. "
"Please re-extract carefully, ensuring that all "
"totals are mathematically correct."
),
})
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
temperature=0.0,
max_tokens=2048,
extra_body={"guided_json": ValidatedInvoice.model_json_schema()},
)
parsed = json.loads(response.choices[0].message.content)
return ValidatedInvoice.model_validate(parsed)
This pattern of extract, validate, and retry with a corrective prompt is one of the most effective techniques for improving extraction accuracy. The retry gives the model a second chance with additional context about what went wrong.
Best Practices for Production Pipelines
Choose the Right Model
Model selection has the largest impact on extraction quality. For straightforward schemas, a 7B to 8B parameter model is often sufficient. For complex documents with nested structures, ambiguous formatting, or domain-specific language, consider a 14B to 32B model. Always benchmark on your actual data before committing. A smaller model that handles your specific document types well is preferable to a larger model that is slower and more expensive to run.
Use Low Temperature for Determinism
Set temperature to 0.0 or a very low value like 0.1 for extraction tasks. You want the model to pick the most likely token at each step, not to be creative. Creativity is the enemy of reliable data extraction.
Always Use Guided Decoding
Never rely on the model to produce valid JSON on its own. Even capable models occasionally produce trailing commas, unquoted keys, or other malformations. Guided decoding eliminates this failure mode entirely and should be considered mandatory for any production extraction pipeline.
Log Everything
Store the raw model output alongside the parsed result for every extraction. When something goes wrong, you need to be able to inspect what the model actually produced. A simple logging approach:
import logging
from pathlib import Path
logger = logging.getLogger("extraction")
logging.basicConfig(level=logging.INFO)
def extract_with_logging(document_text: str, doc_id: str) -> Invoice:
messages = build_extraction_prompt(document_text, Invoice)
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=messages,
temperature=0.0,
max_tokens=2048,
extra_body={"guided_json": Invoice.model_json_schema()},
)
raw_output = response.choices[0].message.content
usage = response.usage
logger.info(
"doc_id=%s prompt_tokens=%d completion_tokens=%d",
doc_id, usage.prompt_tokens, usage.completion_tokens
)
# Persist raw output for debugging
log_path = Path(f"logs/{doc_id}.json")
log_path.parent.mkdir(parents=True, exist_ok=True)
log_path.write_text(raw_output)
return Invoice.model_validate(json.loads(raw_output))
Handle Long Documents with Chunking
If your documents exceed the model's context window, you need a chunking strategy. For extraction, the safest approach is to extract from each chunk independently and then merge results. Be aware that fields like subtotal or total_amount that appear once in the document might appear in multiple chunks, so your merge logic needs to deduplicate intelligently.
Monitor Throughput and Queue Depth
In production, monitor how long requests spend queued before vLLM processes them. If queue times grow, you either need more GPU capacity or need to tune your batch sizes. vLLM exposes Prometheus metrics that make this straightforward to instrument.
Cache Results for Idempotency
Extraction is expensive. If you process the same document twice, you should return the cached result rather than re-running the model. A simple hash of the document text as a cache key works well:
import hashlib
from functools import lru_cache
def document_hash(text: str) -> str:
return hashlib.sha256(text.encode()).hexdigest()
# Use a persistent cache (Redis, SQLite, etc.) in production
_extraction_cache: dict[str, Invoice] = {}
def extract_cached(document_text: str) -> Invoice:
key = document_hash(document_text)
if key in _extraction_cache:
return _extraction_cache[key]
result = extract_document(document_text, Invoice)
_extraction_cache[key] = result
return result
Conclusion
Building a data extraction pipeline with vLLM combines the flexibility of large language models with the reliability of structured output constraints. By leveraging guided decoding to enforce JSON schemas, Pydantic for validation, concurrent batching for throughput, and a retry-with-feedback loop for accuracy, you can create a pipeline that handles real-world document variability at scale. The key insight is that vLLM gives you the performance characteristics of a dedicated parsing service while keeping you in full control of the model and the data. Start with a clear schema, enable guided decoding from day one, validate every output, and iterate on your prompts based on the failure cases you observe in production. With these foundations in place, vLLM makes it practical to turn unstructured documents into clean, structured data that your applications can rely on.