Introduction to Observability and Tracing with MCP
The Model Context Protocol (MCP) has emerged as a standardized way for AI applications to connect with external data sources, tools, and resources. As MCP-based systems grow in complexity—spanning multiple servers, tools, and LLM interactions—understanding what happens inside them becomes critical. This is where observability and tracing come in.
Observability in MCP refers to the ability to inspect, measure, and understand the behavior of MCP clients, servers, and the interactions between them. Tracing, a core pillar of observability, lets you follow a single request as it travels through the protocol's lifecycle—from the initial client call, through tool invocations, resource reads, and prompt generations, all the way back to the response delivered to the user.
In this guide, we'll explore what observability means in the context of MCP, why it matters, how to instrument your MCP servers and clients, and the best practices that will keep your AI systems reliable and debuggable.
What Is Observability in the Context of MCP?
Observability is traditionally defined by three pillars: logs, metrics, and traces. In the MCP world, each pillar takes on a specific flavor:
- Logs: Structured records of discrete events, such as a tool being called, a resource being read, or an error occurring during a prompt rendering.
- Metrics: Numeric measurements over time—tool invocation counts, latency percentiles, error rates, token usage, and active connection counts.
- Traces: A tree of spans representing the full journey of a request across MCP components, including the LLM call, tool execution, and any sub-requests made by tools.
MCP defines its own primitives—tools, resources, prompts, and sampling—that all participate in a typical interaction. A single user message might trigger a tool call, which reads a resource, which itself triggers a sampling request back to the client. Without tracing, this chain of events is opaque. With tracing, you can see exactly where time was spent, where errors originated, and which component is responsible for a degraded experience.
The MCP Lifecycle and Trace Boundaries
A typical MCP interaction flows through several stages, each of which is a candidate for a trace span:
- Client sends a JSON-RPC request to the server (e.g.,
tools/call) - Server resolves the tool handler
- Tool handler executes business logic, possibly calling external APIs
- Tool handler may read MCP resources via
resources/read - Tool handler may request LLM completion via
sampling/createMessage - Server returns the result to the client
- Client feeds the result back into the LLM conversation
Each of these stages should be captured as a span within a single trace, linked by a shared trace context. This allows you to reconstruct the full causal chain after the fact.
Why Observability Matters for MCP Systems
MCP systems are inherently distributed and non-deterministic. An LLM decides which tools to call, tools may call other tools, and sampling requests flow back from server to client. This creates several challenges that observability directly addresses:
Debugging Non-Deterministic Behavior
Unlike traditional APIs where the same input always produces the same output, MCP-mediated interactions depend on LLM reasoning. A tool might work perfectly in testing but fail in production because the LLM called it with unexpected arguments. Traces let you replay the exact sequence of calls and inspect the arguments and results at each step.
Performance Optimization
Latency in MCP systems is often dominated by LLM calls and external tool executions, not the protocol itself. Without tracing, you might assume the MCP server is slow when the real bottleneck is a third-party API called by a tool. Distributed tracing attributes latency to the correct component.
Cost Attribution
LLM calls cost money. When a tool triggers a sampling request, you want to know which user action led to that cost. Traces that include token usage metrics allow you to attribute costs back to specific features or users.
Error Root Cause Analysis
When an MCP tool fails, the error might originate in the tool's internal logic, in an external API, or in the protocol layer. A well-instrumented trace shows the exact span where the error occurred, along with the error message and stack trace, making root cause analysis straightforward.
Core Concepts: Spans, Context, and Propagation
Before diving into implementation, let's establish the vocabulary of distributed tracing as it applies to MCP.
Spans
A span represents a single unit of work. In MCP, a span might be "call tool search_documents" or "read resource file:///data/report.json". Spans have a start time, end time, name, attributes, status, and optionally events (logs attached to the span).
Trace Context and Propagation
A trace is a collection of spans that share a trace ID. When a client calls an MCP server, it should propagate the trace context—typically via headers or metadata—so the server can create child spans within the same trace. This is called context propagation. In MCP, which uses JSON-RPC over stdio, SSE, or WebSocket, propagation is typically done through custom fields in the JSON-RPC message or transport-level headers.
Span Attributes
Attributes are key-value pairs attached to spans. For MCP, useful attributes include:
mcp.method— the JSON-RPC method (e.g.,tools/call)mcp.tool.name— the name of the tool being calledmcp.resource.uri— the URI of the resource being readmcp.server.name— the name of the MCP serverllm.token_count.prompt— prompt tokens consumedllm.token_count.completion— completion tokens produced
Setting Up Tracing in an MCP Server
Let's walk through instrumenting an MCP server using the TypeScript SDK and OpenTelemetry, the de facto standard for distributed tracing. The same concepts apply to the Python SDK.
Installing Dependencies
npm install @modelcontextprotocol/sdk \
@opentelemetry/api \
@opentelemetry/sdk-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/resources \
@opentelemetry/semantic-conventions
Configuring the Tracer Provider
First, set up OpenTelemetry at the entry point of your MCP server. This should happen before any other code runs so that all spans are captured.
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { trace } from "@opentelemetry/api";
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: "mcp-filesystem-server",
[SemanticResourceAttributes.SERVICE_VERSION]: "1.0.0",
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4318/v1/traces",
}),
});
sdk.start();
process.on("SIGTERM", async () => {
await sdk.shutdown();
process.exit(0);
});
export const tracer = trace.getTracer("mcp-filesystem-server", "1.0.0");
Instrumenting Tool Handlers
Now let's create an MCP server with instrumented tool handlers. Each tool call is wrapped in a span that captures the tool name, arguments, duration, and any errors.
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import {
CallToolRequestSchema,
ListToolsRequestSchema,
} from "@modelcontextprotocol/sdk/types.js";
import { tracer } from "./telemetry.js";
import { context, SpanStatusCode } from "@opentelemetry/api";
const server = new Server(
{ name: "mcp-filesystem-server", version: "1.0.0" },
{ capabilities: { tools: {} } }
);
server.setRequestHandler(ListToolsRequestSchema, async () => {
return {
tools: [
{
name: "read_file",
description: "Read the contents of a file",
inputSchema: {
type: "object",
properties: {
path: { type: "string", description: "Absolute file path" },
},
required: ["path"],
},
},
{
name: "search_files",
description: "Search for files matching a pattern",
inputSchema: {
type: "object",
properties: {
pattern: { type: "string", description: "Glob pattern" },
directory: { type: "string", description: "Root directory" },
},
required: ["pattern"],
},
},
],
};
});
server.setRequestHandler(CallToolRequestSchema, async (request) => {
const { name, arguments: args } = request.params;
// Create a span for the tool call
return tracer.startActiveSpan(
`mcp.tool.${name}`,
{
attributes: {
"mcp.method": "tools/call",
"mcp.tool.name": name,
"mcp.tool.arguments": JSON.stringify(args),
},
},
async (span) => {
try {
let result;
switch (name) {
case "read_file":
result = await handleReadFile(args.path, span);
break;
case "search_files":
result = await handleSearchFiles(args.pattern, args.directory, span);
break;
default:
throw new Error(`Unknown tool: ${name}`);
}
span.setAttribute("mcp.tool.success", true);
return result;
} catch (error) {
span.setStatus({
code: SpanStatusCode.ERROR,
message: error.message,
});
span.recordException(error);
span.setAttribute("mcp.tool.success", false);
throw error;
} finally {
span.end();
}
}
);
});
async function handleReadFile(path, parentSpan) {
// Create a child span for the file read operation
return tracer.startActiveSpan("fs.readFile", {
attributes: {
"file.path": path,
},
}, async (span) => {
try {
const fs = await import("fs/promises");
const content = await fs.readFile(path, "utf-8");
span.setAttribute("file.size_bytes", content.length);
return {
content: [{ type: "text", text: content }],
};
} catch (error) {
span.recordException(error);
throw error;
} finally {
span.end();
}
});
}
async function handleSearchFiles(pattern, directory, parentSpan) {
return tracer.startActiveSpan("fs.search", {
attributes: {
"search.pattern": pattern,
"search.directory": directory || ".",
},
}, async (span) => {
try {
const { glob } = await import("glob");
const files = await glob(pattern, {
cwd: directory || ".",
absolute: true,
});
span.setAttribute("search.results_count", files.length);
return {
content: [{ type: "text", text: JSON.stringify(files, null, 2) }],
};
} catch (error) {
span.recordException(error);
throw error;
} finally {
span.end();
}
});
}
const transport = new StdioServerTransport();
await server.connect(transport);
Instrumenting Resource Reads
Resources are another MCP primitive that benefits from tracing. Here's how to instrument a resource handler:
import { ReadResourceRequestSchema } from "@modelcontextprotocol/sdk/types.js";
server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
const uri = request.params.uri;
return tracer.startActiveSpan("mcp.resource.read", {
attributes: {
"mcp.method": "resources/read",
"mcp.resource.uri": uri,
},
}, async (span) => {
try {
const parsed = new URL(uri);
if (parsed.protocol !== "file:") {
throw new Error(`Unsupported protocol: ${parsed.protocol}`);
}
const fs = await import("fs/promises");
const content = await fs.readFile(parsed.pathname, "utf-8");
span.setAttribute("mcp.resource.size_bytes", content.length);
return {
contents: [
{
uri,
mimeType: "text/plain",
text: content,
},
],
};
} catch (error) {
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
span.recordException(error);
throw error;
} finally {
span.end();
}
});
});
Tracing on the MCP Client Side
Observability is only complete when both sides are instrumented. The MCP client—often embedded in an AI application—should create the root span for each interaction and propagate the trace context to the server.
Creating a Traced MCP Client
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import { trace, context, SpanStatusCode, propagation } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";
const tracer = trace.getTracer("mcp-client-app");
propagation.setGlobalPropagator(new W3CTraceContextPropagator());
async function callToolWithTracing(client, toolName, args) {
return tracer.startActiveSpan(
`mcp.client.callTool`,
{
attributes: {
"mcp.tool.name": toolName,
"mcp.tool.arguments": JSON.stringify(args),
},
},
async (span) => {
try {
// Extract current trace context to pass along
const carrier = {};
propagation.inject(context.active(), carrier);
// In a real implementation, you would pass this carrier
// through transport-level headers or JSON-RPC metadata
const result = await client.callTool({
name: toolName,
arguments: args,
});
span.setAttribute("mcp.tool.success", true);
return result;
} catch (error) {
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
span.recordException(error);
throw error;
} finally {
span.end();
}
}
);
}
async function main() {
const transport = new StdioClientTransport({
command: "node",
args: ["./server.js"],
});
const client = new Client(
{ name: "mcp-client-app", version: "1.0.0" },
{ capabilities: {} }
);
await client.connect(transport);
const result = await callToolWithTracing(client, "read_file", {
path: "/tmp/example.txt",
});
console.log(result);
await client.close();
}
main();
Context Propagation Across Transports
The trickiest part of MCP tracing is context propagation. MCP supports multiple transports—stdio, SSE, and WebSocket—and each handles propagation differently:
- stdio: Since stdio doesn't have headers, you can inject trace context into a custom field in the JSON-RPC message, or rely on the fact that client and server run in the same process tree and use a shared exporter.
- SSE / HTTP: Use standard W3C
traceparentandtracestateheaders. Most OpenTelemetry SDKs handle this automatically if you use instrumented HTTP clients. - WebSocket: Inject trace context into the initial handshake headers or into a metadata field of the first JSON-RPC message in a request.
Here's an example of injecting and extracting trace context via JSON-RPC params for stdio transport:
import { propagation, context, trace } from "@opentelemetry/api";
// Client side: inject context into the request
function injectTraceContext(params) {
const carrier = {};
propagation.inject(context.active(), carrier);
return {
...params,
_trace: carrier,
};
}
// Server side: extract context from the request
function extractTraceContext(params) {
if (params._trace) {
return propagation.extract(context.active(), params._trace);
}
return context.active();
}
// Usage in server handler
server.setRequestHandler(CallToolRequestSchema, async (request) => {
const extractedContext = extractTraceContext(request.params);
const activeContext = trace.setSpan(extractedContext, trace.getSpan(extractedContext));
return context.with(activeContext, async () => {
return tracer.startActiveSpan(`mcp.tool.${request.params.name}`, async (span) => {
// ... tool logic
span.end();
});
});
});
Adding Metrics to MCP Servers
While traces give you per-request detail, metrics give you aggregate visibility. Here's how to add key MCP metrics using OpenTelemetry's metrics API.
import { metrics } from "@opentelemetry/api";
const meter = metrics.getMeter("mcp-filesystem-server", "1.0.0");
// Counter: total tool calls
const toolCallCounter = meter.createCounter("mcp.tool.calls", {
description: "Total number of MCP tool calls",
unit: "1",
});
// Histogram: tool call duration
const toolDurationHistogram = meter.createHistogram("mcp.tool.duration", {
description: "Duration of MCP tool calls in milliseconds",
unit: "ms",
});
// Counter: tool errors
const toolErrorCounter = meter.createCounter("mcp.tool.errors", {
description: "Total number of MCP tool errors",
unit: "1",
});
// Observable gauge: active connections
let activeConnections = 0;
meter.createObservableGauge("mcp.connections.active", {
description: "Number of active MCP connections",
}, (observableResult) => {
observableResult.observe(activeConnections);
});
function recordToolCall(toolName, durationMs, success) {
toolCallCounter.add(1, { "mcp.tool.name": toolName });
toolDurationHistogram.record(durationMs, { "mcp.tool.name": toolName });
if (!success) {
toolErrorCounter.add(1, { "mcp.tool.name": toolName });
}
}
You can then call recordToolCall inside your tool handler's finally block to capture metrics alongside traces.
Structured Logging with Trace Correlation
Logs are most powerful when correlated with traces. By including the trace ID and span ID in every log line, you can jump from a trace in your observability dashboard directly to the relevant logs.
import { trace, context } from "@opentelemetry/api";
import winston from "winston";
const logger = winston.createLogger({
level: process.env.LOG_LEVEL || "info",
format: winston.format.combine(
winston.format.timestamp(),
winston.format.json(),
winston.format((info) => {
const span = trace.getSpan(context.active());
if (span) {
const spanContext = span.spanContext();
info.trace_id = spanContext.traceId;
info.span_id = spanContext.spanId;
}
return info;
})()
),
transports: [new winston.transports.Console()],
});
// Usage inside a tool handler
logger.info("Tool call received", {
tool: "read_file",
path: args.path,
});
logger.error("File not found", {
tool: "read_file",
path: args.path,
error: error.message,
});
Observing Sampling Requests
One unique aspect of MCP is the sampling/createMessage request, which flows from server back to client. This reverse-direction call is easy to miss in tracing setups. Make sure to instrument it on both sides:
// Server side: requesting a sample
async function requestSample(client, messages, systemPrompt) {
return tracer.startActiveSpan("mcp.sampling.request", {
attributes: {
"mcp.method": "sampling/createMessage",
"mcp.sampling.message_count": messages.length,
},
}, async (span) => {
try {
const result = await client.request({
method: "sampling/createMessage",
params: {
messages,
systemPrompt,
maxTokens: 1000,
},
});
span.setAttribute("llm.token_count.completion", result.tokenCount || 0);
return result;
} catch (error) {
span.recordException(error);
throw error;
} finally {
span.end();
}
});
}
// Client side: handling a sampling request
server.setRequestHandler("sampling/createMessage", async (request) => {
return tracer.startActiveSpan("mcp.sampling.handle", {
attributes: {
"mcp.method": "sampling/createMessage",
"mcp.sampling.message_count": request.params.messages.length,
},
}, async (span) => {
try {
const result = await callLLM(request.params);
span.setAttribute("llm.token_count.prompt", result.usage.prompt_tokens);
span.setAttribute("llm.token_count.completion", result.usage.completion_tokens);
span.setAttribute("llm.model", result.model);
return result;
} finally {
span.end();
}
});
});
Visualizing MCP Traces
Once you're exporting traces via OTLP, you can visualize them in any OpenTelemetry-compatible backend—Jaeger, Zipkin, Grafana Tempo, Datadog, Honeycomb, or others. A well-instrumented MCP trace will look like a waterfall:
Trace: 4a92f8c1b3e7d6a0
├── mcp.client.callTool (1200ms)
│ ├── mcp.tool.search_files (1180ms)
│ │ ├── fs.search (450ms)
│ │ ├── mcp.resource.read (300ms)
│ │ │ └── fs.readFile (290ms)
│ │ └── mcp.sampling.request (400ms)
│ │ └── mcp.sampling.handle (390ms)
│ │ └── llm.completion (380ms)
│ └── mcp.client.processResult (15ms)
This view immediately tells you that the LLM completion during sampling was the largest contributor to latency, followed by the file system search. Without this trace, you might have blamed the MCP protocol overhead.
Best Practices for MCP Observability
1. Always Propagate Trace Context
The single most important practice is ensuring trace context flows from client to server and back during sampling. Without propagation, you get disconnected spans that are nearly impossible to correlate. Use W3C Trace Context format consistently across all transports.
2. Use Semantic Conventions
Adopt consistent attribute names across your MCP servers. The OpenTelemetry community is developing semantic conventions for GenAI and messaging systems. Follow these patterns and extend them with MCP-specific attributes like mcp.tool.name and mcp.resource.uri.
3. Avoid Sensitive Data in Span Attributes
Tool arguments and resource contents may contain sensitive information. Never log full file contents or user PII as span attributes. Instead, log metadata like file size, MIME type, or a hash of the content. Apply sampling rules to reduce the volume of traces that include detailed attributes.
4. Instrument at Protocol Boundaries
Focus your instrumentation at the boundaries between MCP components—where the client calls the server, where the server calls a tool, where a tool reads a resource, and where a server requests sampling. These boundaries are where context is most likely to be lost and where tracing provides the most value.
5. Capture Token Usage Consistently
Token usage is the primary cost driver in MCP systems. Always capture llm.token_count.prompt and llm.token_count.completion as span attributes and metrics whenever an LLM is called, whether directly by the client or via sampling. This enables cost attribution queries in your observability backend.
6. Set Up Alerting on Key Metrics
Define alerts on the metrics that matter most for MCP systems:
- Tool error rate exceeding 5% over a 5-minute window
- P99 tool call latency exceeding 10 seconds
- Sampling request rate spiking unexpectedly (may indicate a tool loop)
- Token usage per user exceeding daily budgets
7. Use Tail-Based Sampling for Errors
Head-based sampling (deciding whether to sample at span creation time) may miss rare errors. Consider tail-based sampling, which evaluates the complete trace before deciding whether to keep it. This ensures you always capture traces that contain errors or high-latency outliers, while dropping a percentage of healthy traces to manage volume.
8. Version Your Spans
Include the MCP server version and protocol version as resource attributes. When you deploy a new version of a tool, you'll want to compare latency and error rates across versions, which requires version tags on every span.
Putting It All Together: A Complete Instrumented Server
Here's a minimal but complete MCP server with tracing, metrics, and structured logging all wired together:
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import {
CallToolRequestSchema,
ListToolsRequestSchema,
} from "@modelcontextprotocol/sdk/types.js";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { trace, metrics, context, SpanStatusCode } from "@opentelemetry/api";
import winston from "winston";
// --- Telemetry Setup ---
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: "mcp-search-server",
[SemanticResourceAttributes.SERVICE_VERSION]: "2.1.0",
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + "/v1/traces",
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + "/v1/metrics",
}),
exportIntervalMillis: 10000,
}),
});
sdk.start();
process.on("SIGTERM", () => sdk.shutdown().then(() => process.exit(0)));
const tracer = trace.getTracer("mcp-search-server");
const meter = metrics.getMeter("mcp-search-server");
const toolCalls = meter.createCounter("mcp.tool.calls");
const toolErrors = meter.createCounter("mcp.tool.errors");
const toolDuration = meter.createHistogram("mcp.tool.duration", { unit: "ms" });
const logger = winston.createLogger({
level: "info",
format: winston.format.combine(
winston.format.timestamp(),
winston.format.json(),
winston.format((info) => {
const span = trace.getSpan(context.active());
if (span) {
info.trace_id = span.spanContext().traceId;
info.span_id = span.spanContext().spanId;
}
return info;
})()
),
transports: [new winston.transports.Console()],
});
// --- MCP Server ---
const server = new Server(
{ name: "mcp-search-server", version: "2.1.0" },
{ capabilities: { tools: {} } }
);
server.setRequestHandler(ListToolsRequestSchema, async () => ({
tools: [
{
name: "web_search",
description: "Search the web for information",
inputSchema: {
type: "object",
properties: {
query: { type: "string" },
max_results: { type: "number", default: 5 },
},
required: ["query"],
},
},
],
}));
server.setRequestHandler(CallToolRequestSchema, async (request) => {
const { name, arguments: args } = request.params;
const startTime = Date.now();
return tracer.startActiveSpan(`mcp.tool.${name}`, {
attributes: {
"mcp.method": "tools/call",
"mcp.tool.name": name,
},
}, async (span) => {
try {
logger.info("Tool call started", { tool: name, args });
let result;
if (name === "web_search") {
result = await performSearch(args.query, args.max_results || 5, span);
} else {
throw new Error(`Unknown tool: ${name}`);
}
const durationMs = Date.now() - startTime;
toolCalls.add(1, { "mcp.tool.name": name, "mcp.tool.status": "success" });
toolDuration.record(durationMs, { "mcp.tool.name": name });
span.setAttribute("mcp.tool.duration_ms", durationMs);
logger.info("Tool call completed", { tool: name, durationMs });
return result;
} catch (error) {
const durationMs = Date.now() - startTime;
toolErrors.add(1, { "mcp.tool.name": name });
toolDuration.record(durationMs, { "mcp.tool.name": name });
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
span.recordException(error);
logger.error("Tool call failed", { tool: name, error: error.message, durationMs });
throw error;
} finally {
span.end();
}
});
});
async function performSearch(query, maxResults, parentSpan) {
return tracer.startActiveSpan("web.search", {
attributes: {
"search.query": query,
"search.max_results": maxResults,
},
}, async (span) => {
try {
// Simulate an external API call
const response = await fetch(
`https://api.example.com/search?q=${encodeURIComponent(query)}&limit=${maxResults}`
);
if (!response.ok) {
throw new Error(`Search API returned ${response.status}`);
}
const data = await response.json();
span.setAttribute("search.results_count", data.results.length);
return {
content: [{
type: "text",
text: JSON.stringify(data.results, null, 2),
}],
};
} finally {
span.end();
}
});
}
const transport = new StdioServerTransport();
await server.connect(transport);
logger.info("MCP server started", { name: "mcp-search-server", version: "2.1.0" });
Conclusion
Observability and tracing are not optional for production-grade MCP systems—they are the foundation upon which reliability, performance, and cost management are built. By instrumenting both your MCP clients and servers with OpenTelemetry traces, metrics, and structured logs, you gain end-to-end visibility into the complex dance of tool calls, resource reads, and sampling requests that characterize modern AI applications. The key takeaways are to propagate trace context across every protocol boundary, use consistent semantic conventions for MCP-specific attributes, capture token usage for cost attribution, and correlate logs with traces for fast debugging. Start with the instrumentation patterns shown in this guide, adapt them to your specific tools and resources, and iterate as your MCP ecosystem grows. The investment in observability pays off the first time you need to explain why a tool call took 30 seconds or why an LLM started calling the wrong tool in production.