Detecting Hallucinations in Production LLM Outputs

Detecting Hallucinations in Production LLM Outputs

Large language models are powerful, but they share one dangerous trait: they can produce fluent, confident, and entirely fabricated information. In production systems—where outputs may drive medical advice, financial decisions, or customer-facing content—hallucinations are not just an inconvenience. They are a liability. This tutorial walks through what hallucinations are, why detecting them matters, and how to build practical detection pipelines you can deploy today.

What Is an LLM Hallucination?

A hallucination occurs when a language model generates text that is syntactically correct and contextually plausible but factually wrong, ungrounded, or fabricated. Hallucinations come in several flavors:

• Factual hallucinations: The model states something objectively false (e.g., "Abraham Lincoln was born in 1950").
• Contextual hallucinations: The model contradicts the source context it was given, even if the statement might be true in isolation.
• Logical hallucinations: The model makes reasoning errors, such as incorrect arithmetic or flawed deductions.
• Citation hallucinations: The model invents references, URLs, or quotes that do not exist.

In a retrieval-augmented generation (RAG) setup, the most common and dangerous type is the contextual hallucination: the model drifts away from the retrieved documents and fills gaps with parametric memory.

Why Hallucination Detection Matters in Production

During prototyping, a hallucination might be a funny anecdote. In production, it becomes a business risk. Here is why detection is essential:

• Trust and safety: Users rely on outputs. Wrong answers erode trust and can cause real harm.
• Compliance: Regulated industries (healthcare, finance, legal) require demonstrable accuracy.
• Cost control: Catching bad outputs before they reach users prevents expensive downstream remediation.
• Observability: You cannot improve what you cannot measure. Detection gives you metrics to track model quality over time.
• Guardrails for autonomous agents: Agents that take actions based on LLM outputs need verification layers to avoid catastrophic decisions.

Approaches to Hallucination Detection

There is no single silver bullet. Production systems typically combine multiple techniques. Below are the most practical approaches, with code you can adapt.

1. Self-Consistency Checking

The idea is simple: ask the model the same question multiple times (or rephrase it) and compare the answers. If the model gives contradictory responses, the answer is likely ungrounded. This works well for factual queries.

import openai
from collections import Counter

openai.api_key = "your-api-key"

def generate(prompt, temperature=0.7):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return response.choices[0].message.content.strip()

def self_consistency_check(question, n=5):
    answers = [generate(question) for _ in range(n)]
    
    # Normalize and count answers
    normalized = [a.lower().strip() for a in answers]
    counts = Counter(normalized)
    most_common, freq = counts.most_common(1)[0]
    
    agreement = freq / n
    return {
        "answers": answers,
        "agreement_score": agreement,
        "is_consistent": agreement >= 0.8,
    }

result = self_consistency_check("What year was the Eiffel Tower completed?")
print(f"Agreement: {result['agreement_score']:.0%}")
print(f"Consistent: {result['is_consistent']}")

The agreement score tells you how stable the answer is. Low agreement signals low confidence and a higher hallucination risk. The tradeoff is cost: you are calling the model n times per query.

2. LLM-as-a-Judge Verification

Use a second model (or the same model with a different prompt) to verify whether the generated answer is supported by the provided context. This is one of the most popular production techniques because it is flexible and relatively cheap.

VERIFIER_PROMPT = """You are a strict fact-checker.

Given a SOURCE CONTEXT and a CLAIM, determine whether the claim
is fully supported by the source context.

Rules:
- If the claim contains any fact not found in the source context,
  mark it as "UNSUPPORTED".
- If the claim contradicts the source context, mark it as
  "CONTRADICTED".
- Only mark "SUPPORTED" if every factual assertion in the claim
  can be traced to the source context.

Source Context:
{context}

Claim:
{claim}

Respond in JSON:
{{"verdict": "SUPPORTED|UNSUPPORTED|CONTRADICTED", "reason": "..."}}
"""

def verify_claim(context, claim):
    prompt = VERIFIER_PROMPT.format(context=context, claim=claim)
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        response_format={"type": "json_object"},
    )
    import json
    return json.loads(response.choices[0].message.content)

context = """
The Eiffel Tower was completed in 1889 for the World's Fair
celebrating the 100th anniversary of the French Revolution.
It stands 330 meters tall.
"""

claim = "The Eiffel Tower was completed in 1889 and is 330 meters tall."
result = verify_claim(context, claim)
print(result)
# {'verdict': 'SUPPORTED', 'reason': 'Both facts appear in the context.'}

bad_claim = "The Eiffel Tower was completed in 1889 and was designed by Gustave Eiffel."
result = verify_claim(context, bad_claim)
print(result)
# {'verdict': 'UNSUPPORTED', 'reason': 'The designer is not mentioned in the context.'}

This approach is especially powerful in RAG pipelines. You pass the retrieved chunks as context and the model's answer as the claim. The verifier acts as a gatekeeper before the response reaches the user.

3. Token-Level Confidence via Logprobs

Some API providers return log probabilities for generated tokens. Low logprob values indicate the model was uncertain. While not a direct hallucination signal, aggregated low-confidence tokens correlate with ungrounded content.

def generate_with_logprobs(prompt):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        logprobs=True,
        top_logprobs=1,
    )
    return response.choices[0]

def confidence_score(choice):
    import math
    logprobs = choice.logprobs.content
    if not logprobs:
        return None
    
    probabilities = []
    for token in logprobs:
        if token.logprob is not None:
            probabilities.append(math.exp(token.logprob))
    
    if not probabilities:
        return None
    
    # Geometric mean of token probabilities
    log_sum = sum(math.log(p) for p in probabilities)
    return math.exp(log_sum / len(probabilities))

choice = generate_with_logprobs("What is the capital of France?")
score = confidence_score(choice)
print(f"Confidence: {score:.4f}")
print(f"Answer: {choice.message.content}")

if score and score < 0.3:
    print("WARNING: Low confidence — possible hallucination")

Use this as a signal, not a verdict. A model can be confidently wrong, so pair logprob analysis with content-based verification for robust detection.

4. NLI-Based Grounding Detection

Natural Language Inference (NLI) models classify the relationship between a premise (your source context) and a hypothesis (the model's claim) as entailment, contradiction, or neutral. This is a fast, cheap, and deterministic alternative to LLM-as-a-judge.

from transformers import pipeline

# Load an NLI model fine-tuned for fact-checking / grounding
nli = pipeline(
    "text-classification",
    model="roberta-large-mnli",
    return_all_scores=True,
)

def check_grounding(context, claim):
    # NLI models expect premise-hypothesis pairs
    result = nli(f"{context}  {claim}")
    scores = {r["label"].lower(): r["score"] for r in result[0]}
    
    if scores.get("entailment", 0) > 0.7:
        return "GROUNDED", scores
    elif scores.get("contradiction", 0) > 0.7:
        return "CONTRADICTED", scores
    else:
        return "UNVERIFIED", scores

context = "The Apollo 11 mission landed on the Moon on July 20, 1969."
claim = "Apollo 11 landed on the Moon in 1969."

label, scores = check_grounding(context, claim)
print(f"Label: {label}")
print(f"Scores: {scores}")

NLI models are lightweight enough to run on CPU and can process thousands of claims per second on a modest GPU. This makes them ideal for high-throughput production environments where calling a second LLM for every response is too expensive.

5. Building a Combined Detection Pipeline

No single method catches everything. A robust production system layers multiple checks. Here is a simplified pipeline that combines NLI grounding, confidence scoring, and LLM verification:

import math
import json
from transformers import pipeline

nli = pipeline("text-classification", model="roberta-large-mnli",
               return_all_scores=True)

class HallucinationDetector:
    def __init__(self, nli_threshold=0.7, confidence_threshold=0.3):
        self.nli_threshold = nli_threshold
        self.confidence_threshold = confidence_threshold
    
    def check_nli(self, context, claim):
        result = nli(f"{context}  {claim}")
        scores = {r["label"].lower(): r["score"] for r in result[0]}
        return scores
    
    def check_confidence(self, choice):
        logprobs = choice.logprobs.content if choice.logprobs else None
        if not logprobs:
            return 1.0  # assume confident if unavailable
        probs = [math.exp(t.logprob) for t in logprobs if t.logprob]
        if not probs:
            return 1.0
        log_sum = sum(math.log(p) for p in probs)
        return math.exp(log_sum / len(probs))
    
    def llm_verify(self, context, claim):
        prompt = f"""Is this claim fully supported by the context?
Context: {context}
Claim: {claim}
Respond JSON: {{"supported": true/false, "reason": "..."}}"""
        resp = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0,
            response_format={"type": "json_object"},
        )
        return json.loads(resp.choices[0].message.content)
    
    def evaluate(self, context, claim, choice=None):
        nli_scores = self.check_nli(context, claim)
        confidence = self.check_confidence(choice) if choice else 1.0
        
        # Fast path: strong NLI signal
        if nli_scores.get("entailment", 0) > self.nli_threshold:
            return {"verdict": "PASS", "method": "nli", "scores": nli_scores}
        
        if nli_scores.get("contradiction", 0) > self.nli_threshold:
            return {"verdict": "FAIL", "method": "nli", "scores": nli_scores}
        
        # Slow path: fall back to LLM verification
        llm_result = self.llm_verify(context, claim)
        verdict = "PASS" if llm_result["supported"] else "FAIL"
        
        return {
            "verdict": verdict,
            "method": "llm_verify",
            "nli_scores": nli_scores,
            "confidence": confidence,
            "llm_reason": llm_result.get("reason"),
        }

# Usage
detector = HallucinationDetector()
context = "Photosynthesis converts light energy into chemical energy stored in glucose."
claim = "Photosynthesis produces glucose from light energy."
result = detector.evaluate(context, claim)
print(json.dumps(result, indent=2))

This pipeline uses cheap NLI checks first and only escalates to an LLM call when the NLI model is uncertain. This keeps latency and cost low while maintaining accuracy.

Best Practices for Production

Log Everything for Observability

Store every input, output, context, and detection result. Over time, this data becomes a goldmine for improving your system and training custom classifiers.

import logging
from datetime import datetime

logger = logging.getLogger("hallucination_monitor")

def log_evaluation(query, context, output, result):
    logger.info(json.dumps({
        "timestamp": datetime.utcnow().isoformat(),
        "query": query,
        "context": context[:500],
        "output": output[:500],
        "verdict": result["verdict"],
        "method": result.get("method"),
        "scores": result.get("scores") or result.get("nli_scores"),
    }))

Set Thresholds Based on Use Case

A medical chatbot needs stricter thresholds than a creative writing assistant. Tune your nli_threshold and confidence_threshold based on the cost of false negatives versus false positives. Measure precision and recall against a labeled evaluation set.

Handle Failures Gracefully

When detection flags a potential hallucination, do not silently fail. Options include:

• Returning a fallback message like "I'm not confident in this answer."
• Triggering a retry with a more constrained prompt.
• Escalating to a human reviewer for high-stakes queries.
• Providing the source context alongside the answer so users can verify.

Use Specialized Tools and Frameworks

Several open-source libraries are purpose-built for this:

• Ragas: Evaluates RAG pipelines, including faithfulness and answer relevance metrics.
• DeepEval: Pytest-like framework for LLM evaluation with hallucination metrics.
• Guardrails AI: Adds validators to LLM outputs, including fact-checking guards.
• TruLens: Observability and evaluation for LLM apps with groundedness tracking.

# Example using Ragas for faithfulness evaluation
from ragas import evaluate
from ragas.metrics import faithfulness
from datasets import Dataset

eval_data = Dataset.from_dict({
    "question": ["What is the speed of light?"],
    "answer": ["The speed of light is 299,792,458 meters per second."],
    "contexts": [["Light travels at 299,792,458 m/s in a vacuum."]],
})

results = evaluate(eval_data, metrics=[faithfulness])
print(results)
# {'faithfulness': 1.0}

Continuously Evaluate and Retune

Hallucination patterns shift as you change prompts, models, or retrieval strategies. Set up a weekly evaluation job that runs your detector against a golden dataset and tracks metrics over time. Alert on regressions.

Conclusion

Detecting hallucinations in production LLM outputs is not a one-time fix but an ongoing discipline. By combining self-consistency checks, LLM-as-a-judge verification, logprob confidence signals, and NLI-based grounding detection, you can build a layered defense that catches most fabrications before they reach users. The key is to treat hallucination detection as a first-class part of your architecture—with its own logging, metrics, thresholds, and fallback strategies—rather than an afterthought. Start simple with an NLI grounding check, measure its performance on your real traffic, and iteratively add layers until your false-negative rate meets the risk tolerance of your application. Your users, your compliance team, and your future self will thank you.