LangSmith for Agent Debugging: Complete Setup Guide

What is LangSmith?

LangSmith is a developer platform built by the creators of LangChain to help you debug, test, evaluate, and monitor LLM-powered applications—especially agents. It gives you full observability over every step your application takes, from the initial prompt to the final output, including all intermediate tool calls, retries, and reasoning steps. LangSmith automatically captures traces and logs of your LangChain (or custom) runs, providing a rich UI and a programmatic client to inspect, filter, and analyze those traces.

Why LangSmith Matters for Agent Debugging

Debugging AI agents is fundamentally different from debugging traditional software. Agents are non-deterministic, often involve multiple LLM calls, external tool invocations, and complex decision graphs. A single error might occur deep inside a chain after several successful steps, or an agent might silently produce a suboptimal answer due to a flawed reasoning loop. Without proper observability, you’re left guessing. LangSmith solves this by:

Full traceability — every LLM call, tool execution, retry, and state change is logged.
Visual timeline — see exactly when a step happened, how long it took, and what inputs/outputs were.
Error pinpointing — immediately locate which node in the chain failed and why.
Metadata and tagging — attach user IDs, release versions, or experiment tags to filter and compare runs.
Cost and latency tracking — identify bottlenecks and expensive calls.

Getting Started: Setup and Configuration

To begin debugging agents with LangSmith, you need an account, the Python SDK, and a few environment variables. This guide assumes you're using LangChain (v0.1+), but LangSmith also works with custom orchestration code.

1. Create an Account and Get an API Key

Visit smith.langchain.com and sign up (free for personal use with generous limits). Navigate to Settings → API Keys and create a key. You'll need this key to send traces.

2. Install Dependencies

Install the LangChain core library and the LangSmith SDK. The SDK is included with langchain but can also be installed separately for more advanced usage.

pip install langchain langchain-openai langsmith

3. Configure Environment Variables

Set the following environment variables in your development environment (or in a .env file). These tell LangChain to send traces to LangSmith and identify your project.

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY="your-api-key-here"
export LANGCHAIN_PROJECT="my-agent-project"  # optional, defaults to 'default'

For production, you might want to set LANGCHAIN_ENDPOINT if using a self-hosted instance. The LANGCHAIN_PROJECT helps organize traces; use different project names for different environments (e.g., production, staging).

4. Enable Tracing in Code (Implicit or Explicit)

With the environment variables set, any LangChain run (e.g., agent.invoke()) is automatically traced. You don’t need to modify your agent code. However, you can also manually create a LangSmith client for custom logging or feedback.

from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain.tools import tool
from langchain_core.prompts import ChatPromptTemplate

# Define a simple tool
@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

tools = [multiply]
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}")
])

agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# This single call will automatically generate a trace in LangSmith
result = executor.invoke({"input": "What is 23 times 17?"})
print(result['output'])

After running this, go to your LangSmith dashboard. You’ll see a trace named something like AgentExecutor with a complete breakdown: the LLM call, tool invocation, and final answer.

Inspecting Traces and Debugging Agent Runs

The LangSmith UI is the primary debugging surface. Open any trace and you’ll see a tree view of the run steps. Each step shows:

Input and output data
Latency (time spent in that step)
Token usage (for LLM calls)
Status (success, error, or retry)
Any metadata you attached

For an agent, you can drill into:

Prompt formatting — see exactly what the LLM received, including the system message and tool definitions.
LLM reasoning — the raw completion output, often including a thought or tool selection.
Tool execution — inputs passed to your tool and the returned result.
Agent scratchpad updates — how the agent accumulated intermediate steps.

This makes it trivial to spot mistakes: e.g., the LLM chose a wrong tool name, a tool returned an unexpected format, or the agent looped endlessly. You can also compare runs side-by-side.

Programmatic Access with the Client SDK

Beyond the UI, you can query and manipulate traces using the langsmith Python client. This is powerful for automated analysis, reporting, or adding feedback.

from langsmith import Client

client = Client()

# List recent runs in your project
runs = list(client.list_runs(
    project_name="my-agent-project",
    execution_order=1,  # root runs only
    start_time="2025-01-01T00:00:00Z"
))

for run in runs[:3]:
    print(f"Run ID: {run.id}, Status: {run.status}, Output: {run.outputs}")

# Get a specific trace and inspect child runs
trace = client.read_run(run_id="your-run-id")
child_runs = list(client.list_runs(parent_run_id=trace.id))
for child in child_runs:
    print(f"Child run name: {child.name}, error: {child.error}")

Advanced Debugging Techniques

LangSmith goes beyond passive tracing. You can actively improve your agent by incorporating feedback, running evaluations, and using datasets for regression testing.

Adding Custom Metadata and Tags

Attach metadata like user IDs, deployment version, or experiment identifiers to each run. This lets you filter traces in the UI and compare performance across versions.

from langsmith import traceable
import uuid

# Decorate your function to send traces with custom metadata
@traceable(
    run_type="tool",
    name="custom_search",
    metadata={"version": "1.2.3", "env": "staging"}
)
def custom_search(query: str) -> str:
    # Your tool implementation
    return f"Results for {query}"

# Or add tags to the run inside your agent logic
from langchain.callbacks import LangSmithRunManager

# Within an agent callback or chain, you can get the current run ID
# and attach tags via the client
client = Client()
run_id = ...  # obtain from context or callback
client.update_run(run_id, tags=["beta", "user-feedback-trial"])

Human Feedback and Run Evaluation

You can log feedback (e.g., thumbs up/down, ratings) directly on a trace to build a dataset of good vs. bad runs. This is essential for iterating on prompts or agent logic.

# Send feedback on a specific run
client.create_feedback(
    run_id="your-run-id",
    key="user_rating",
    score=1,  # 1 for positive, 0 for negative
    comment="The agent used the correct tool and gave a clear answer."
)

# Or create a feedback from an automated evaluator
client.create_feedback(
    run_id="your-run-id",
    key="correctness",
    score=0.85,
    source="auto-evaluator-v1"
)

Using Datasets and Testing

LangSmith allows you to create datasets from real runs (or manually) and then run evaluations against those datasets. This is critical for catching regressions when you change a prompt or tool.

# Create a dataset from existing runs
dataset = client.create_dataset(
    dataset_name="multiplication_tasks",
    description="A collection of multiplication queries"
)

# Add examples from historical runs
for run in runs:
    client.create_example(
        dataset_id=dataset.id,
        inputs={"input": run.inputs["input"]},
        outputs={"output": run.outputs["output"]}
    )

# Later, run an evaluation
from langsmith.evaluation import evaluate

evaluate(
    dataset_name="multiplication_tasks",
    target=executor.invoke,  # your agent executor function
    evaluators=[your_custom_evaluator],  # functions that compare output
    project_name="eval-run-001"
)

Best Practices for Agent Debugging with LangSmith

Always use project names — separate traces by environment (dev, staging, prod) and feature branch to avoid noise.
Tag liberally — add tags for user IDs, session IDs, experiment names, or error categories. This makes filtering in the UI fast.
Attach metadata at the start — include version info, model name, and any context that helps later analysis.
Set up automated feedback — even a simple “completed successfully” heuristic helps build a dataset of healthy runs.
Review traces regularly — not just when something breaks. Spot trends like increasing latency or rising token counts.
Version your agents and prompts — use LangSmith’s traceable decorator with version metadata to compare performance across iterations.
Integrate evaluations into CI — run LangSmith evaluations as part of your test suite before deploying an agent update.

Conclusion

LangSmith transforms agent debugging from a painful, log-grepping chore into a structured, visual, and programmable process. By automatically capturing every detail of your agent’s execution, providing a rich UI for inspection, and offering APIs for feedback and testing, it enables you to understand exactly what your agent is doing—and why. Start with the simple environment setup, let your existing LangChain agents generate traces, and gradually adopt tagging, feedback, and dataset evaluations to build a robust development loop. As agents grow more autonomous and complex, this kind of observability isn’t optional—it’s essential.