What Are Multi-Agent Debate Systems?
Multi-Agent Debate Systems are an architectural pattern where multiple Large Language Model (LLM) instances—or "agents"—are prompted to take on distinct perspectives and engage in structured argumentation around a given problem. Rather than relying on a single model call, the system orchestrates a back-and-forth exchange where agents challenge each other's reasoning, point out flaws, and iteratively refine a final answer.
At its core, the pattern works like this:
- Proposer Agent — generates an initial solution or reasoning chain
- Opponent / Critic Agent — identifies weaknesses, logical gaps, or factual errors in the proposer's output
- Moderator / Judge Agent — synthesizes the arguments from both sides and produces a final, improved answer
This structure can be extended to N agents with specialized roles—mathematical verifier, ethics checker, domain expert, adversarial tester—all debating concurrently or in rounds. The key insight is that reasoning improves through adversarial collaboration, much like red-teaming in security or peer review in academia.
Why Multi-Agent Debate Matters
Single-model reasoning suffers from well-documented failure modes:
- Confirmation bias — the model latches onto its first answer and rationalizes backward
- Overconfidence in wrong answers — especially dangerous in math, logic, and medical domains
- Shallow reasoning chains — the model stops thinking once it produces a plausible-sounding result
- Hallucination cascades — one invented fact leads to a chain of fabricated supporting evidence
Debate systems address these by forcing contradiction into the loop. When Agent B explicitly tries to break Agent A's argument, both agents must engage in deeper reasoning. Research from organizations like Anthropic, DeepMind, and OpenAI has shown that debate protocols can significantly improve truth-seeking behavior, reduce hallucination rates, and produce more calibrated uncertainty estimates.
The technique matters because it provides a training-free improvement layer—you don't need to fine-tune the underlying model. You simply orchestrate existing models more intelligently. This makes it immediately applicable to any LLM API (GPT-4, Claude, Gemini, open-source models via vLLM, etc.).
Core Architecture: A Practical Implementation
Let's build a complete multi-agent debate system from scratch using Python and the OpenAI API. The system will have three agents—Proposer, Critic, and Judge—that debate a given question for multiple rounds.
import os
import json
from openai import OpenAI
from typing import List, Dict, Optional
from dataclasses import dataclass, field
import time
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
@dataclass
class DebateRound:
"""Tracks the state of a single debate round."""
round_number: int
proposer_response: str
critic_response: str
judge_synthesis: Optional[str] = None
@dataclass
class DebateConfig:
"""Configuration for the debate system."""
question: str
num_rounds: int = 2
model: str = "gpt-4o"
temperature: float = 0.7
proposer_role: str = "expert_solver"
critic_role: str = "skeptical_reviewer"
judge_role: str = "neutral_arbiter"
max_tokens_per_response: int = 800
verbose: bool = True
class MultiAgentDebateSystem:
"""
A multi-agent debate system that improves LLM reasoning through
structured adversarial collaboration.
"""
def __init__(self, config: DebateConfig):
self.config = config
self.debate_history: List[DebateRound] = []
self.system_prompts = self._build_system_prompts()
def _build_system_prompts(self) -> Dict[str, str]:
"""Construct role-specific system prompts for each agent."""
proposer_prompt = f"""You are the PROPOSER agent in a debate. Your role is to provide the most
accurate, well-reasoned answer to the user's question.
Guidelines:
- Break down complex problems step by step (Chain of Thought)
- Cite relevant facts, principles, or data when applicable
- Express confidence levels honestly (e.g., "I am 90% confident because...")
- Anticipate potential objections and address them preemptively
- If uncertain about something, state it explicitly rather than guessing
- Your answer must stand up to rigorous critical scrutiny"""
critic_prompt = f"""You are the CRITIC agent in a debate. Your role is to rigorously examine
the Proposer's answer and identify any weaknesses.
Guidelines:
- Look for logical fallacies, missing steps, or unwarranted assumptions
- Check for factual errors or outdated information
- Identify edge cases where the reasoning breaks down
- Question confidence levels that seem over- or under-confident
- Suggest specific improvements rather than just vague criticism
- Be adversarial but constructive—your goal is better truth, not just winning
- If the Proposer's answer is genuinely excellent, acknowledge that honestly"""
judge_prompt = f"""You are the JUDGE agent in a debate. Your role is to synthesize the arguments
from both the Proposer and the Critic into a final, improved answer.
Guidelines:
- Weigh both arguments objectively based on evidence and logic
- Incorporate valid criticisms and discard invalid ones
- Produce a final answer that is more accurate than either individual argument
- Explain your reasoning for accepting or rejecting specific points
- If consensus is reached, state it clearly; if not, explain the remaining uncertainty
- Format your output with: FINAL ANSWER, followed by DETAILED REASONING"""
return {
"proposer": proposer_prompt,
"critic": critic_prompt,
"judge": judge_prompt
}
def _call_agent(self, system_prompt: str, user_message: str,
agent_name: str) -> str:
"""Make an API call to an agent and return its response."""
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}
]
if self.config.verbose:
print(f"\n{'='*60}")
print(f"🤖 Calling {agent_name} agent...")
print(f"{'='*60}")
try:
response = client.chat.completions.create(
model=self.config.model,
messages=messages,
temperature=self.config.temperature,
max_tokens=self.config.max_tokens_per_response
)
content = response.choices[0].message.content
if self.config.verbose:
print(f"\n{content[:300]}..." if len(content) > 300 else f"\n{content}")
return content
except Exception as e:
print(f"Error calling {agent_name}: {e}")
raise
def run_debate(self) -> Dict:
"""
Execute the full multi-round debate and return results.
"""
question = self.config.question
# Round 1: Initial proposal and first criticism
for round_num in range(1, self.config.num_rounds + 1):
if self.config.verbose:
print(f"\n{'🔵'*30}")
print(f"DEBATE ROUND {round_num}")
print(f"{'🔵'*30}")
# Build context from previous rounds
context = self._build_debate_context(current_round=round_num)
# Proposer generates or refines answer
proposer_message = f"""QUESTION: {question}
PREVIOUS DEBATE CONTEXT:
{context}
Please provide your {'revised ' if round_num > 1 else ''}answer to the question.
{'Address the criticisms from previous rounds and strengthen your reasoning.' if round_num > 1 else ''}"""
proposer_response = self._call_agent(
self.system_prompts["proposer"],
proposer_message,
"PROPOSER"
)
# Critic examines the proposer's response
critic_message = f"""QUESTION: {question}
PROPOSER'S ANSWER TO CRITIQUE:
{proposer_response}
PREVIOUS DEBATE CONTEXT:
{context}
Please provide your detailed critique of the Proposer's answer above."""
critic_response = self._call_agent(
self.system_prompts["critic"],
critic_message,
"CRITIC"
)
# Store the round
debate_round = DebateRound(
round_number=round_num,
proposer_response=proposer_response,
critic_response=critic_response
)
self.debate_history.append(debate_round)
# Brief pause to avoid rate limits
time.sleep(0.5)
# Final: Judge synthesizes everything
judge_message = self._build_judge_message(question)
judge_response = self._call_agent(
self.system_prompts["judge"],
judge_message,
"JUDGE"
)
# Attach judge response to last round
self.debate_history[-1].judge_synthesis = judge_response
return {
"question": question,
"debate_rounds": [
{
"round": r.round_number,
"proposer": r.proposer_response,
"critic": r.critic_response,
"judge_synthesis": r.judge_synthesis
}
for r in self.debate_history
],
"final_answer": judge_response,
"config": {
"model": self.config.model,
"num_rounds": self.config.num_rounds,
"temperature": self.config.temperature
}
}
def _build_debate_context(self, current_round: int) -> str:
"""Construct context string from previous debate rounds."""
if not self.debate_history:
return "No previous rounds. This is the opening argument."
context_parts = []
for debate_round in self.debate_history:
context_parts.append(f"""
--- ROUND {debate_round.round_number} ---
PROPOSER: {debate_round.proposer_response}
CRITIC: {debate_round.critic_response}
""")
return "\n".join(context_parts)
def _build_judge_message(self, question: str) -> str:
"""Build the comprehensive message for the Judge agent."""
debate_transcript = self._build_debate_context(
current_round=self.config.num_rounds + 1
)
return f"""QUESTION: {question}
COMPLETE DEBATE TRANSCRIPT:
{debate_transcript}
You must now deliver your FINAL JUDGMENT. Synthesize all arguments.
Output format:
FINAL ANSWER:
[Your synthesized, improved answer]
DETAILED REASONING:
[Explain which arguments you accepted, which you rejected, and why]
"""
# Example usage
config = DebateConfig(
question="If a bat and a ball cost $1.10 total, and the bat costs $1.00 more than the ball, how much does the ball cost?",
num_rounds=2,
model="gpt-4o",
temperature=0.7,
verbose=True
)
debate_system = MultiAgentDebateSystem(config)
result = debate_system.run_debate()
print("\n" + "="*60)
print("FINAL SYNTHESIZED ANSWER:")
print("="*60)
print(result["final_answer"])Understanding the Cognitive Safety Net
The example above uses a deliberately tricky question—the classic bat-and-ball problem that reliably triggers the "fast thinking" error (answering 10 cents instead of 5 cents). Single-model calls frequently get this wrong because the intuitive-but-wrong answer feels satisfying. The debate system forces a second look.
Here's what happens under the hood during a typical debate round:
- Round 1 — Proposer may fall for the intuitive trap and answer $0.10. Critic points out: "If the ball is $0.10, then the bat at $1.00 more would be $1.10, making the total $1.20—which contradicts the given total of $1.10. Let's solve algebraically..."
- Round 2 — Proposer acknowledges the error, shows the algebraic solution: let ball = x, bat = x + 1.00, total = x + (x + 1.00) = 2x + 1.00 = 1.10, so 2x = 0.10, x = 0.05. Critic verifies: ball = $0.05, bat = $1.05, sum = $1.10 ✓
- Judge — Confirms the correct answer, notes where the initial reasoning failed, and delivers a calibrated final response.
Advanced Pattern: Specialized Agent Roles
For complex domains, generic "proposer vs. critic" isn't enough. You can assign specialized roles that each examine different facets of the problem. Here's an expanded system with five agents:
class SpecializedDebateSystem:
"""
A multi-agent debate system with domain-specialized agents
that examine different facets of a problem.
"""
def __init__(self, question: str, model: str = "gpt-4o"):
self.question = question
self.model = model
self.agents = {
"logical_reasoner": {
"system_prompt": """You are a LOGICAL REASONER. Your job is to analyze
problems using formal logic, identify premises, evaluate deductive and inductive
arguments, and spot logical fallacies. Use step-by-step reasoning.""",
"response": None
},
"fact_checker": {
"system_prompt": """You are a FACT CHECKER. Your job is to verify
factual claims, check for accuracy against known information, and flag any
statements that are unverifiable or likely false. Be precise about sources.""",
"response": None
},
"mathematical_verifier": {
"system_prompt": """You are a MATHEMATICAL VERIFIER. Your job is to
check all calculations, ensure numerical reasoning is sound, verify equations,
and catch arithmetic errors. Show your verification work.""",
"response": None
},
"edge_case_analyzer": {
"system_prompt": """You are an EDGE CASE ANALYZER. Your job is to
stress-test solutions by considering extreme scenarios, boundary conditions,
and unusual inputs that might break the reasoning.""",
"response": None
},
"synthesizer": {
"system_prompt": """You are a SYNTHESIZER. Your job is to read all
analyses from specialized agents, reconcile contradictions, and produce a final
answer that incorporates all verified insights.""",
"response": None
}
}
def run_analysis(self) -> Dict:
"""Run all specialized agents and synthesize results."""
# Phase 1: Parallel specialized analysis
analysis_results = {}
for agent_name, agent_info in self.agents.items():
if agent_name == "synthesizer":
continue # Runs last
response = self._call_agent(
agent_info["system_prompt"],
f"Please analyze this problem thoroughly: {self.question}",
agent_name
)
self.agents[agent_name]["response"] = response
analysis_results[agent_name] = response
time.sleep(0.3) # Rate limit buffer
# Phase 2: Synthesizer receives all analyses
analyses_text = "\n\n".join([
f"=== {name.upper()} ANALYSIS ===\n{resp}"
for name, resp in analysis_results.items()
])
synthesizer_prompt = f"""
All specialized agents have analyzed this question:
QUESTION: {self.question}
{analyses_text}
Synthesize all findings into a comprehensive final answer.
Resolve any contradictions. Highlight remaining uncertainties.
Structure: FINAL ANSWER, then SUPPORTING EVIDENCE from each agent."""
final_response = self._call_agent(
self.agents["synthesizer"]["system_prompt"],
synthesizer_prompt,
"SYNTHESIZER"
)
self.agents["synthesizer"]["response"] = final_response
return {
"question": self.question,
"specialist_analyses": analysis_results,
"synthesized_answer": final_response
}
def _call_agent(self, system_prompt: str, user_message: str,
agent_name: str) -> str:
response = client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}
],
temperature=0.7,
max_tokens=600
)
return response.choices[0].message.content
# Usage
specialized_system = SpecializedDebateSystem(
question="A pharmaceutical company claims their new drug reduces symptoms in 75% of patients. " +
"A clinical trial with 200 patients showed 140 improved. Is the company's claim statistically " +
"justified at a 95% confidence level?",
model="gpt-4o"
)
result = specialized_system.run_analysis()
print(result["synthesized_answer"])Integrating with LangChain and Tool Use
For production systems, you'll often want agents to access external tools—calculators, search APIs, code interpreters, or databases. Here's how to wire a debate system into LangChain with tool-calling capabilities:
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain_core.messages import SystemMessage, HumanMessage, ToolMessage
from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated, Sequence
import operator
import json
# Define tools that agents can use
@tool
def calculator(expression: str) -> str:
"""Evaluate a mathematical expression. Use for all calculations."""
try:
# Safe eval with limited namespace
result = eval(expression, {"__builtins__": {}},
{"abs": abs, "round": round, "sum": sum,
"pow": pow, "sqrt": lambda x: x**0.5})
return str(result)
except Exception as e:
return f"Error: {str(e)}"
@tool
def web_search(query: str) -> str:
"""Search for factual information. Returns relevant snippets."""
# In production, wire this to a real search API
return f"Search results for '{query}': [Simulated search results would appear here]"
class DebateState(TypedDict):
question: str
messages: Annotated[list, operator.add]
round: int
proposer_tools_called: list
critic_tools_called: list
final_answer: str
def build_tool_enriched_debate_graph():
"""
Build a LangGraph debate system where agents can invoke
calculators, search engines, and other tools mid-debate.
"""
llm = ChatOpenAI(model="gpt-4o", temperature=0.7)
tools = [calculator, web_search]
llm_with_tools = llm.bind_tools(tools)
def proposer_node(state: DebateState):
"""Proposer agent that can use tools to strengthen arguments."""
system_msg = SystemMessage(content="""You are the PROPOSER in a debate.
You have access to a calculator and web search. Use them to verify your
reasoning before presenting arguments. Show your work.""")
context = f"QUESTION: {state['question']}\n"
if state['round'] > 0:
context += f"Previous round criticisms are in the message history."
messages = [system_msg, HumanMessage(content=context)]
response = llm_with_tools.invoke(messages)
# Handle tool calls if the LLM requests them
if hasattr(response, 'tool_calls') and response.tool_calls:
tool_messages = []
for tool_call in response.tool_calls:
tool_name = tool_call['name']
tool_args = tool_call['args']
if tool_name == 'calculator':
result = calculator.invoke(tool_args['expression'])
elif tool_name == 'web_search':
result = web_search.invoke(tool_args['query'])
tool_messages.append(
ToolMessage(content=result, tool_call_id=tool_call['id'])
)
state['proposer_tools_called'].append({
'tool': tool_name,
'args': tool_args,
'result': result
})
# Re-invoke with tool results
all_messages = messages + [response] + tool_messages
final_response = llm.invoke(all_messages)
state['messages'].append(final_response.content)
return state
def critic_node(state: DebateState):
"""Critic that fact-checks using tools."""
system_msg = SystemMessage(content="""You are the CRITIC. Use tools to
verify the Proposer's claims. Check calculations and search for factual
contradictions.""")
# Similar tool-calling pattern as proposer
proposer_text = state['messages'][-1] if state['messages'] else ""
messages = [system_msg, HumanMessage(
content=f"Critique this argument:\n{proposer_text}"
)]
response = llm_with_tools.invoke(messages)
# ... handle tool calls similarly ...
state['messages'].append(response.content)
state['round'] += 1
return state
def judge_node(state: DebateState):
"""Final synthesis after debate rounds."""
system_msg = SystemMessage(content="You are the JUDGE. Deliver final answer.")
debate_history = "\n".join(state['messages'])
messages = [system_msg, HumanMessage(
content=f"Question: {state['question']}\nDebate:\n{debate_history}\n\nDeliver final synthesis."
)]
response = llm.invoke(messages)
state['final_answer'] = response.content
return state
# Build the graph
graph = StateGraph(DebateState)
graph.add_node("proposer", proposer_node)
graph.add_node("critic", critic_node)
graph.add_node("judge", judge_node)
graph.set_entry_point("proposer")
graph.add_edge("proposer", "critic")
graph.add_edge("critic", "proposer") # Loop for multiple rounds
# In production, add conditional edges based on round count
graph.add_edge("proposer", "judge") # After rounds complete
return graph.compile()
# The graph can be invoked with:
# debate_graph = build_tool_enriched_debate_graph()
# result = debate_graph.invoke({"question": "...", "round": 0, ...})Debate Protocols: Designing the Interaction Rules
The structure of the debate—who speaks when, what they see, and how consensus is reached—dramatically affects outcomes. Here are the key protocol variants:
- Sequential Turn-Taking — Proposer → Critic → Proposer → Critic → Judge. Simple and effective. Each agent sees the full history.
- Blind Critique — The Critic receives the Proposer's answer but NOT the Proposer's reasoning chain. This prevents the Critic from being persuaded by plausible-sounding but flawed logic. The Critic must independently identify problems.
- Parallel Specialist + Synthesizer — Multiple agents analyze independently (no cross-talk), then a synthesizer reconciles. Prevents groupthink but may miss interactive synergy.
- Iterative Refinement with Voting — N agents each propose answers, then vote on the best. The winning answer is refined by a designated improver agent. Effective for creative tasks.
- Adversarial Zero-Shot — The same model plays both roles but with different system prompts. Surprisingly effective because the model's "critic persona" can catch errors its "proposer persona" made.
Practical Code: Blind Critique Protocol
The blind critique pattern is particularly powerful for catching reasoning errors. Here's the implementation:
class BlindCritiqueDebate:
"""
Implements a blind critique protocol where the Critic sees the answer
but NOT the reasoning chain, forcing independent verification.
"""
def __init__(self, question: str, model: str = "gpt-4o"):
self.question = question
self.model = model
def _extract_final_answer(self, full_response: str) -> str:
"""Extract just the conclusion from a detailed reasoning chain."""
# In production, use more sophisticated parsing
extraction_prompt = f"""
Given the following detailed response, extract ONLY the final conclusion
or answer. Strip all reasoning, explanation, and commentary.
Return just the bare answer:
RESPONSE:
{full_response}
EXTRACTED ANSWER:"""
response = client.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": extraction_prompt}],
temperature=0.0, # Deterministic extraction
max_tokens=200
)
return response.choices[0].message.content.strip()
def run_blind_debate(self, num_rounds: int = 2) -> Dict:
"""Execute debate where critic sees only conclusions."""
proposer_system = """You are a careful reasoner. For each problem:
1. Work through your reasoning step by step in detail
2. At the very end, state your FINAL ANSWER clearly marked with ###FINAL_ANSWER###
3. Keep your reasoning and final answer clearly separated"""
critic_system = """You are a CRITIC. You will be shown ONLY the final answer
(not the reasoning). You must:
1. Assess whether the answer is correct based on your own reasoning
2. If you suspect an error, explain what the correct answer should be
3. Do NOT assume the hidden reasoning is correct just because the answer looks plausible"""
rounds_data = []
previous_critic_feedback = ""
for round_num in range(1, num_rounds + 1):
# Proposer: full reasoning + answer
proposer_msg = f"""QUESTION: {self.question}
PREVIOUS CRITIC FEEDBACK: {previous_critic_feedback or 'None'}
Provide your full reasoning and final answer."""
proposer_full = self._call_chat(proposer_system, proposer_msg)
final_answer_only = self._extract_final_answer(proposer_full)
# Critic: sees ONLY the final answer
critic_msg = f"""QUESTION: {self.question}
THE PROPOSER'S FINAL ANSWER (reasoning hidden): {final_answer_only}
Critique this answer. Is it correct? What should the answer be?"""
critic_full = self._call_chat(critic_system, critic_msg)
previous_critic_feedback = critic_full
rounds_data.append({
"round": round_num,
"proposer_full_reasoning": proposer_full,
"extracted_answer_shown_to_critic": final_answer_only,
"critic_blind_critique": critic_full
})
# Judge sees everything
judge_msg = f"""QUESTION: {self.question}
FULL DEBATE RECORD (Judge sees all reasoning):
{json.dumps(rounds_data, indent=2)}
Provide final synthesis with FINAL ANSWER and reasoning."""
final_judgment = self._call_chat(
"You are a JUDGE synthesizing a debate. Deliver the most accurate answer.",
judge_msg
)
return {
"question": self.question,
"rounds": rounds_data,
"final_judgment": final_judgment
}
def _call_chat(self, system_prompt: str, user_message: str) -> str:
response = client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message}
],
temperature=0.7,
max_tokens=800
)
return response.choices[0].message.content
# Example: Testing the blind critique on a reasoning-heavy problem
blind_debate = BlindCritiqueDebate(
question="""A tank fills from a pipe at 10 gallons per minute.
Simultaneously, water drains from a hole at the bottom at a rate equal to
5% of the current volume per minute. If the tank starts empty and has a
capacity of 1000 gallons, how long until it's full?""",
model="gpt-4o"
)
result = blind_debate.run_blind_debate(num_rounds=2)
print(result["final_judgment"])Best Practices for Production Deployments
After building numerous debate systems, several patterns emerge for getting reliable results:
- Start with a strong single-model baseline — Measure accuracy, confidence calibration, and error modes of a single LLM call. The debate system should improve on this baseline measurably. If your single model already achieves 99% accuracy on a task, the debate overhead may not be worth it.
- Calibrate temperature per agent — Use temperature 0.0–0.3 for fact-checkers, mathematical verifiers, and judges (you want deterministic verification). Use 0.5–0.8 for proposers and creative brainstormers. A critic with temperature 0.0 may miss novel failure modes that a slightly more exploratory critic (0.3–0.5) would catch.
-
Implement early termination — If the Critic responds with "The Proposer's answer is completely correct and I have no criticisms," route directly to the Judge. Don't force artificial debate rounds that might introduce errors into a correct answer. Here's a simple check:
def critic_is_satisfied(critic_response: str) -> bool: satisfaction_markers = [ "no criticisms", "completely correct", "fully accurate", "i agree entirely", "no errors found", "answer is correct" ] response_lower = critic_response.lower() return any(marker in response_lower for marker in satisfaction_markers) - Log everything for evaluation — Store full debate transcripts with timestamps, model versions, and token counts. You'll need this data to measure whether the debate actually improved accuracy (did the final answer change from wrong to right?) and to debug regression cases (did debate make a correct answer worse?).
- Use different models for different agents — A common pattern is using a fast, cheap model (GPT-3.5, Claude Haiku) for the Critic and a more capable model (GPT-4, Claude Opus) for the Proposer. Or vice versa—sometimes a weaker Proposer + strong Critic produces better results because the strong model catches more errors.
- Guard against "debate collapse" — Sometimes agents converge on a wrong answer because the Critic's argument sounds convincing even when incorrect. Mitigate this by having the Judge re-verify key claims independently, or by adding a "devil's advocate" round where agents must argue the opposite position.
- Budget token usage carefully — A 3-round debate with 3 agents can consume 5–10× the tokens of a single call. For high-volume applications, consider: (a) using smaller context windows per agent, (b) summarizing previous rounds rather than passing full history, (c) running debate only on high-stakes or low-confidence queries.
- Parallelize where possible — In the specialist pattern, all specialist agents can run simultaneously (no dependencies). This cuts wall-clock latency dramatically. Only the synthesizer must wait for all results.
Measuring Debate Effectiveness
To know whether your debate system actually improves reasoning, you need rigorous evaluation. Here's a framework for measuring impact:
class DebateEvaluator:
"""
Quantitatively measures whether multi-agent debate improves
accuracy over single-model baselines.
"""
def __init__(self, test_dataset: List[Dict]):
"""
test_dataset: List of {"question": str, "ground_truth": str,
"category": str, "difficulty": str}
"""
self.dataset = test_dataset
self.results = []
def evaluate(self, debate_system_class, single_model_func):
"""
Compare debate system against single-model performance.
"""
for item in self.dataset:
question = item["question"]
ground_truth = item["ground_truth"]
# Single model baseline
single_answer = single_model_func(question)
single_correct = self._judge_correctness(
single_answer, ground_truth
)
# Debate system
debate_config = DebateConfig(
question=question,
num_rounds=2,
model="gpt-4o",
verbose=False
)
debate_system = debate_system_class(debate_config)
debate_result = debate_system.run_debate()
debate_answer = debate_result["final_answer"]
debate_correct = self._judge_correctness(
debate_answer, ground_truth
)
self.results.append({
"question": question,
"category": item.get("category", "general"),
"difficulty": item.get("difficulty", "medium"),
"single_model_correct": single_correct,
"debate_correct": debate_correct,
"debate_fixed_error": (not single_correct and debate_correct),
"debate_introduced_error": (single_correct and not debate_correct),
"single_answer": single_answer,
"debate_answer": debate_answer,
"ground_truth": ground_truth
})
return self._compute_metrics()
def _judge_correctness(self, answer: str, ground_truth: str) -> bool:
"""Use an LLM judge to evaluate correctness."""
judge_prompt = f"""
Compare the given answer to the ground truth.
Answer TRUE if the given answer is substantively correct
(equivalent in meaning, even if wording differs).
Answer FALSE if it contradicts the ground truth or is incorrect.
GIVEN ANSWER: {answer}
GROUND TRUTH: {ground_truth}
CORRECT (TRUE/FALSE):"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": judge_prompt}],
temperature=0.0,
max_tokens=10
)
return "TRUE" in response.choices