Building a Knowledge Base Chatbot with OpenAI Agents SDK: Complete Guide

Building a Knowledge Base Chatbot with OpenAI Agents SDK: Complete Guide

Knowledge base chatbots have become essential tools for customer support, internal documentation access, and educational platforms. With the release of the OpenAI Agents SDK, developers now have a powerful framework for building intelligent agents that can reason, use tools, and maintain context across conversations. This guide walks you through building a production-ready knowledge base chatbot from scratch using the OpenAI Agents SDK.

What Is the OpenAI Agents SDK?

The OpenAI Agents SDK is a Python framework designed to simplify the construction of autonomous AI agents. Unlike the raw Chat Completions API, the Agents SDK provides abstractions for agents, tools, handoffs, guardrails, and tracing. It allows you to define agents with specific instructions, equip them with callable tools, and orchestrate multi-agent workflows.

A knowledge base chatbot built on this SDK can do more than retrieve documents. It can reason about which documents to search, synthesize answers from multiple sources, cite sources, and gracefully handle follow-up questions. The SDK handles the conversation loop, tool execution, and context management for you.

Why Use the Agents SDK for a Knowledge Base Chatbot?

Traditional retrieval-augmented generation (RAG) pipelines require significant glue code. You must manage embeddings, vector search, prompt construction, and response formatting manually. The Agents SDK reduces this complexity in several ways:

• Tool abstraction: Define a search function once, and the agent decides when and how to call it.
• Built-in conversation loop: The SDK automatically manages multi-turn interactions and tool call sequences.
• Structured outputs: Enforce response schemas so your chatbot returns consistent, parseable answers.
• Tracing and observability: Every agent run is traced, making debugging and evaluation straightforward.
• Guardrails: Validate inputs and outputs to prevent misuse or hallucination outside your knowledge domain.

Prerequisites and Setup

Before writing code, ensure you have Python 3.9 or higher installed. You will also need an OpenAI API key with access to the models used by the SDK. For the knowledge base itself, this tutorial uses a simple in-memory vector store, but the same patterns apply to Pinecone, Weaviate, or pgvector.

Install the required packages:

pip install openai-agents openai numpy

Set your API key as an environment variable:

export OPENAI_API_KEY="sk-your-api-key-here"

Step 1: Building the Knowledge Base

The knowledge base is the foundation of your chatbot. It stores documents, generates embeddings, and performs semantic search. For this tutorial, we will build a lightweight vector store using NumPy. In production, you would replace this with a dedicated vector database.

import numpy as np
from openai import OpenAI

client = OpenAI()

class KnowledgeBase:
    def __init__(self):
        self.documents = []
        self.embeddings = []

    def add_document(self, title: str, content: str, source: str = ""):
        """Add a document to the knowledge base with its embedding."""
        text = f"{title}\n{content}"
        embedding = self._get_embedding(text)
        self.documents.append({
            "title": title,
            "content": content,
            "source": source,
        })
        self.embeddings.append(embedding)

    def _get_embedding(self, text: str) -> list:
        response = client.embeddings.create(
            model="text-embedding-3-small",
            input=text,
        )
        return response.data[0].embedding

    def search(self, query: str, top_k: int = 3) -> list:
        """Return the top_k most relevant documents for the query."""
        if not self.documents:
            return []

        query_embedding = np.array(self._get_embedding(query))
        doc_embeddings = np.array(self.embeddings)

        # Cosine similarity
        similarities = doc_embeddings.dot(query_embedding) / (
            np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(query_embedding)
        )

        top_indices = np.argsort(similarities)[::-1][:top_k]
        results = []
        for idx in top_indices:
            doc = self.documents[idx]
            results.append({
                "title": doc["title"],
                "content": doc["content"],
                "source": doc["source"],
                "score": float(similarities[idx]),
            })
        return results

Now populate the knowledge base with sample documents. In a real application, you would load these from files, databases, or APIs.

kb = KnowledgeBase()

kb.add_document(
    title="Return Policy",
    content="Customers can return any item within 30 days of purchase for a full refund. Items must be in original packaging with receipt. Refunds are processed within 5-7 business days.",
    source="policies.md",
)

kb.add_document(
    title="Shipping Information",
    content="Standard shipping takes 3-5 business days and costs $5.99. Express shipping takes 1-2 business days and costs $14.99. Free shipping is available on orders over $50.",
    source="shipping.md",
)

kb.add_document(
    title="Warranty Coverage",
    content="All electronic products come with a 1-year manufacturer warranty covering defects in materials and workmanship. Accidental damage is not covered. Extended warranties are available for purchase.",
    source="warranty.md",
)

kb.add_document(
    title="Account Security",
    content="Users should enable two-factor authentication for account security. Passwords must be at least 12 characters long. Suspicious activity can be reported to security@example.com.",
    source="security.md",
)

Step 2: Creating the Search Tool

The Agents SDK uses the concept of tools to extend an agent's capabilities. A tool is simply a Python function decorated with @function_tool. The agent can call this tool autonomously when it determines that searching the knowledge base is necessary.

from agents import function_tool

@function_tool
def search_knowledge_base(query: str) -> str:
    """Search the company knowledge base for relevant information.
    
    Args:
        query: The search query describing what information is needed.
    
    Returns:
        A formatted string containing the most relevant documents found.
    """
    results = kb.search(query, top_k=3)
    
    if not results:
        return "No relevant documents found in the knowledge base."
    
    formatted = []
    for i, doc in enumerate(results, 1):
        formatted.append(
            f"Document {i}: {doc['title']}\n"
            f"Source: {doc['source']}\n"
            f"Relevance Score: {doc['score']:.2f}\n"
            f"Content: {doc['content']}\n"
        )
    
    return "\n".join(formatted)

The docstring is critical. The SDK uses it to help the model understand when and how to use the tool. Write clear, descriptive docstrings that explain the tool's purpose and parameters.

Step 3: Defining the Agent

Now create the agent itself. An agent is defined by its name, model, instructions, and tools. The instructions act as the system prompt, shaping the agent's behavior and personality.

from agents import Agent

kb_agent = Agent(
    name="KnowledgeBaseAssistant",
    model="gpt-4o",
    instructions=(
        "You are a helpful customer support assistant for an e-commerce company. "
        "Your job is to answer user questions using the company knowledge base.\n\n"
        "Rules:\n"
        "1. Always use the search_knowledge_base tool to find relevant information before answering.\n"
        "2. Base your answers only on the information returned by the tool. Do not make up facts.\n"
        "3. If the knowledge base does not contain relevant information, say so honestly.\n"
        "4. Cite the source document when providing information.\n"
        "5. Be concise but complete in your responses.\n"
        "6. If a user asks a follow-up question, search again to ensure accuracy.\n"
    ),
    tools=[search_knowledge_base],
)

Step 4: Running the Chatbot

With the agent defined, you can now run it. The SDK provides a Runner class that executes the agent and manages the conversation loop, including tool calls and responses.

import asyncio
from agents import Runner

async def chat():
    print("Knowledge Base Chatbot (type 'quit' to exit)\n")
    
    conversation_history = []
    
    while True:
        user_input = input("You: ").strip()
        
        if user_input.lower() in ("quit", "exit", "q"):
            print("Goodbye!")
            break
        
        if not user_input:
            continue
        
        # Run the agent with conversation context
        result = await Runner.run(
            starting_agent=kb_agent,
            input=user_input,
            context=conversation_history,
        )
        
        print(f"\nAssistant: {result.final_output}\n")
        
        # Store the exchange for context
        conversation_history.append({"role": "user", "content": user_input})
        conversation_history.append({"role": "assistant", "content": result.final_output})

if __name__ == "__main__":
    asyncio.run(chat())

When you run this script, the agent will receive your question, decide to call the search tool, process the results, and return a grounded answer. Here is an example interaction:

You: How long do I have to return a product?

Assistant: According to our Return Policy (source: policies.md), customers can return any item within 30 days of purchase for a full refund. Items must be in original packaging with a receipt, and refunds are processed within 5-7 business days.

You: Do you offer free shipping?

Assistant: Yes, free shipping is available on orders over $50, as stated in our Shipping Information document (source: shipping.md). Standard shipping otherwise costs $5.99 and takes 3-5 business days, while express shipping costs $14.99 and takes 1-2 business days.

Step 5: Adding Structured Output

For many applications, you want the chatbot to return structured data rather than free-form text. The Agents SDK supports Pydantic models for output validation. This is useful when your chatbot feeds into another system or when you want to display sources separately from the answer.

from pydantic import BaseModel, Field

class ChatbotResponse(BaseModel):
    answer: str = Field(description="The answer to the user's question")
    sources: list[str] = Field(description="List of source documents cited")
    confidence: str = Field(description="Confidence level: high, medium, or low")
    follow_up_suggestions: list[str] = Field(
        description="Suggested follow-up questions the user might ask"
    )

structured_agent = Agent(
    name="StructuredKBAssistant",
    model="gpt-4o",
    instructions=(
        "You are a customer support assistant. Use the search_knowledge_base tool "
        "to find information, then provide a structured response. "
        "Set confidence to 'low' if no relevant documents were found. "
        "Always cite source documents in the sources field."
    ),
    tools=[search_knowledge_base],
    output_type=ChatbotResponse,
)

Running this agent returns a validated ChatbotResponse object:

result = await Runner.run(starting_agent=structured_agent, input="What does the warranty cover?")

response = result.final_output
print(f"Answer: {response.answer}")
print(f"Sources: {response.sources}")
print(f"Confidence: {response.confidence}")
print(f"Follow-ups: {response.follow_up_suggestions}")

Step 6: Implementing Guardrails

Guardrails validate inputs and outputs to keep your chatbot within its intended scope. For a knowledge base chatbot, you might want to reject questions that are off-topic or potentially harmful. The SDK lets you define guardrail functions that run before or after the main agent.

from agents import GuardrailFunctionOutput, input_guardrail

@input_guardrail
async def topic_guardrail(ctx, agent, input_data):
    """Reject questions that are clearly outside customer support scope."""
    off_topic_keywords = ["politics", "religion", "stock market", "medical advice"]
    
    input_lower = input_data.lower() if isinstance(input_data, str) else ""
    
    for keyword in off_topic_keywords:
        if keyword in input_lower:
            return GuardrailFunctionOutput(
                output_info={"reason": f"Topic '{keyword}' is out of scope"},
                tripwire_triggered=True,
            )
    
    return GuardrailFunctionOutput(
        output_info={"reason": "Topic is within scope"},
        tripwire_triggered=False,
    )

guarded_agent = Agent(
    name="GuardedKBAssistant",
    model="gpt-4o",
    instructions=kb_agent.instructions,
    tools=[search_knowledge_base],
    input_guardrails=[topic_guardrail],
)

When a user asks an off-topic question, the guardrail trips and the agent does not process the request. You can catch this in your application code and return a polite refusal.

Step 7: Multi-Agent Handoffs

As your knowledge base grows, a single agent may struggle to handle all domains effectively. The SDK supports handoffs, allowing one agent to transfer control to another specialized agent. For example, you could have separate agents for shipping, returns, and technical support.

shipping_agent = Agent(
    name="ShippingSpecialist",
    model="gpt-4o",
    instructions=(
        "You are a shipping specialist. Answer questions about shipping rates, "
        "delivery times, and tracking. Use the search_knowledge_base tool."
    ),
    tools=[search_knowledge_base],
)

returns_agent = Agent(
    name="ReturnsSpecialist",
    model="gpt-4o",
    instructions=(
        "You are a returns and refunds specialist. Answer questions about return "
        "policies, refund processing, and exchanges. Use the search_knowledge_base tool."
    ),
    tools=[search_knowledge_base],
)

triage_agent = Agent(
    name="TriageAgent",
    model="gpt-4o",
    instructions=(
        "You are a triage agent. Determine which specialist should handle the user's "
        "question and hand off to the appropriate agent. If the question is general, "
        "answer it directly using the search_knowledge_base tool."
    ),
    tools=[search_knowledge_base],
    handoffs=[shipping_agent, returns_agent],
)

Now run the triage agent. It will automatically route conversations to the right specialist:

result = await Runner.run(
    starting_agent=triage_agent,
    input="I want to return a defective phone. How do I do that?",
)
print(result.final_output)

Best Practices

Write Detailed Tool Docstrings

The model relies on tool descriptions to decide when to call them. Vague docstrings lead to poor tool selection. Include the purpose, parameter meanings, and return format in every docstring.

Chunk Documents Effectively

Large documents should be split into meaningful chunks before embedding. A chunk size of 500-1000 tokens with some overlap typically works well. Each chunk should be self-contained enough to be useful on its own.

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list:
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    start = 0
    
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start = end - overlap
    
    return chunks

Use the Right Embedding Model

The text-embedding-3-small model is cost-effective for most use cases. For higher accuracy, especially with technical or domain-specific content, consider text-embedding-3-large. Benchmark both on your data before committing.

Implement Caching

Embedding generation is the most expensive operation in a knowledge base chatbot. Cache embeddings to avoid regenerating them for the same documents. A simple dictionary cache works for small bases; Redis is better for production.

from functools import lru_cache

@lru_cache(maxsize=10000)
def get_cached_embedding(text: str) -> tuple:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return tuple(response.data[0].embedding)

Monitor and Evaluate with Tracing

The SDK's tracing feature logs every agent run, including tool calls, intermediate steps, and token usage. Review traces regularly to identify cases where the agent calls the wrong tool, retrieves irrelevant documents, or hallucinates.

from agents import enable_verbose_stdout_logging

enable_verbose_stdout_logging()

Set Temperature Appropriately

For factual knowledge base responses, use a low temperature to reduce randomness. The Agents SDK allows you to configure model settings per agent:

from agents import ModelSettings

factual_agent = Agent(
    name="FactualAssistant",
    model="gpt-4o",
    model_settings=ModelSettings(temperature=0.2, max_tokens=500),
    instructions="Answer factually using only the knowledge base.",
    tools=[search_knowledge_base],
)

Handle Edge Cases Gracefully

Not every question will have an answer in your knowledge base. Train your agent to say "I don't know" rather than fabricating information. This is primarily controlled through instructions, but you can also add an output guardrail to detect unsupported claims.

Conclusion

Building a knowledge base chatbot with the OpenAI Agents SDK gives you a robust, maintainable foundation for intelligent document retrieval and question answering. By combining semantic search tools, well-crafted agent instructions, structured outputs, guardrails, and multi-agent handoffs, you can create a chatbot that is both accurate and scalable. Start with the simple single-agent setup described in this guide, then gradually add structure and specialization as your knowledge base and user needs grow. The SDK's tracing and observability features will help you iterate confidently, ensuring your chatbot delivers reliable, grounded answers at every step.