← Back to DevBytes
Go July 17, 2026 16 min read

MongoDB API Performance: Profiling and Optimization

Understanding MongoDB Performance Profiling

MongoDB performance profiling is the systematic process of collecting, analyzing, and interpreting query execution data to identify bottlenecks, slow-running operations, and inefficient query patterns. At its core, profiling gives you a window into exactly how MongoDB executes your queries — which indexes are used, how many documents are scanned, and where time is actually spent.

When you enable profiling, MongoDB records detailed metadata about every operation that meets your threshold criteria. This metadata includes the operation type, execution time in milliseconds, the query shape, the plan chosen by the query optimizer, and the number of documents examined versus returned. The profiler stores this data in a capped system collection called system.profile, which lives inside each database you profile.

What Gets Captured in a Profile Entry

A typical profile document contains these critical fields:

The ratio between docsExamined and nreturned is perhaps the most telling signal of inefficiency. When MongoDB scans thousands of documents but returns only a handful, you have a clear indexing opportunity.

Why Performance Profiling Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Unoptimized queries don't just slow down a single endpoint — they create compounding effects across your entire application. Here's why profiling deserves dedicated attention in your development workflow:

Profiling transforms opaque slowness into concrete, measurable problems you can fix. Without it, you're debugging blind.

Enabling and Configuring the MongoDB Profiler

The profiler offers three verbosity levels that balance insight against overhead:

Level 2 is invaluable during development and debugging but imposes measurable overhead in production. Level 1 with a sensible threshold (typically 100–200ms) provides targeted signal without excessive noise.

Enabling the Profiler via the Shell

// Enable profiling at level 1 with a 100ms threshold on the current database
db.setProfilingLevel(1, { slowms: 100 })

// Check current profiling status
db.getProfilingStatus()
// Output: { "was": 0, "slowms": 100, "sampleRate": 1, "ok": 1 }

// Enable level 2 for comprehensive capture (development only)
db.setProfilingLevel(2)

Enabling Profiling via the MongoDB Driver (Node.js Example)

const { MongoClient } = require('mongodb');

async function enableProfiling() {
  const client = new MongoClient('mongodb://localhost:27017');
  await client.connect();
  
  const adminDb = client.db('admin');
  
  // Set profiling on the 'mydb' database: level 1, 100ms threshold
  const result = await adminDb.command({
    profile: 1,
    slowms: 100
  }, { dbName: 'mydb' });
  
  console.log('Profiler enabled:', result);
  
  // Verify the setting
  const status = await adminDb.command({
    profile: -1
  }, { dbName: 'mydb' });
  
  console.log('Profiler status:', status);
  await client.close();
}

enableProfiling();

Setting a Sample Rate for High-Volume Collections

For extremely high-throughput collections, profiling every slow query can itself become expensive. MongoDB 4.0+ supports a sample rate that profiles only a percentage of slow operations:

// Profile only 25% of slow queries to reduce profiler overhead
db.setProfilingLevel(1, { slowms: 100, sampleRate: 0.25 })

The sample rate is a floating-point value between 0 and 1. A rate of 0.1 means approximately 10% of qualifying operations get logged. This is particularly useful on sharded clusters where profiler overhead multiplies across nodes.

Querying and Analyzing Profile Data

The system.profile collection is a regular capped collection that you can query just like any other. Here are practical analysis queries you can run immediately:

Finding the Slowest Recent Queries

// Top 10 slowest queries in the last hour, sorted by execution time
db.system.profile.find({
  ts: { $gte: new Date(Date.now() - 3600000) },
  op: 'query'
}).sort({ millis: -1 }).limit(10).pretty()

Identifying Collection Scans (Missing Indexes)

// Find queries that performed full collection scans
db.system.profile.find({
  op: 'query',
  'command.collection': 'users',
  planSummary: { $regex: /COLLSCAN/ }
}).sort({ millis: -1 }).limit(20)

Aggregating Slow Query Patterns

// Group slow queries by namespace to identify problematic collections
db.system.profile.aggregate([
  { $match: { millis: { $gte: 100 }, op: 'query' } },
  { $group: {
      _id: '$ns',
      avgMillis: { $avg: '$millis' },
      maxMillis: { $max: '$millis' },
      count: { $sum: 1 },
      totalTime: { $sum: '$millis' }
  }},
  { $sort: { totalTime: -1 } },
  { $limit: 10 }
])

Examining a Specific Slow Query in Detail

// Retrieve the full command object for a suspicious query
db.system.profile.findOne({ millis: { $gte: 500 } }).command

The command field reveals the exact filter, sort order, and projection the client sent. You can copy this object directly into an explain() call to reproduce and analyze the query plan.

Using the Explain Method for Deep Query Analysis

The profiler tells you that a query was slow; explain() tells you why. This method runs the query planner without executing the full operation and returns the chosen execution plan with granular stage-level timing.

Explain Verbosity Modes

Running Explain on a Suspect Query

// Full executionStats analysis for a problematic find query
db.users.explain('executionStats').find({
  email: { $regex: /@gmail\.com$/ },
  status: 'active',
  created_at: { $gte: new Date('2024-01-01') }
}).sort({ last_login: -1 }).limit(50)

Interpreting the Explain Output

{
  executionStats: {
    executionSuccess: true,
    nReturned: 50,
    executionTimeMillis: 1420,
    totalKeysExamined: 0,        // No index used — red flag
    totalDocsExamined: 850000,   // Scanned 850K documents to return 50
    executionStages: {
      stage: 'SORT',             // In-memory sort after collection scan
      sortExecutes: 1,
      inputStage: {
        stage: 'COLLSCAN',       // Full collection scan
        filter: { ... },
        docsExamined: 850000
      }
    }
  }
}

Key warning signs in explain output:

Common Performance Problems and Their Fixes

Problem 1: Missing Index Leading to Collection Scan

// Before: COLLSCAN on 1 million documents, 800ms execution
db.orders.explain('executionStats').find({
  customer_id: 'CUST-12345',
  status: 'shipped'
})

// Fix: Create a compound index covering the filter fields
db.orders.createIndex({ customer_id: 1, status: 1 })

// After: IXSCAN, 2ms execution, only 15 keys examined

Problem 2: Index Not Covering Sort Field

// Before: Uses index on {created_at: 1} but then performs in-memory SORT
// on the 'total' field, examining 10K documents, 450ms
db.invoices.explain('executionStats').find({
  created_at: { $gte: new Date('2024-06-01') }
}).sort({ total: -1 }).limit(20)

// Fix: Compound index that includes the sort field
db.invoices.createIndex({ created_at: 1, total: -1 })

// After: Index provides both filtering and sort order, 3ms execution

Problem 3: Regex Leading Index Scan Without Prefix

// Before: Index on {email: 1} exists but regex doesn't use a prefix
// MongoDB scans all index entries matching the regex (slow)
db.users.find({ email: /@gmail\.com$/ })  // No prefix — inefficient

// Fix: Use a caret-anchored regex for index prefix matching
db.users.find({ email: /^customer.*@gmail\.com$/ })  // Uses index prefix

// Alternative: Store a separate normalized field for efficient querying
// Add an 'email_domain' field and create an index on it
db.users.createIndex({ email_domain: 1 })
db.users.find({ email_domain: 'gmail.com' })  // Simple equality, fast index lookup

Problem 4: Large Skip Values on Sorted Queries

// Before: Skip(5000).limit(20) forces MongoDB to scan 5020 index entries
// Execution time grows linearly with skip value
db.posts.find({ author: 'jdoe' })
  .sort({ created_at: -1 })
  .skip(5000)
  .limit(20)

// Fix: Use range-based pagination with a cursor on the sort field
const lastCreatedAt = new Date('2024-11-15T10:30:00Z');
db.posts.find({
  author: 'jdoe',
  created_at: { $lt: lastCreatedAt }
}).sort({ created_at: -1 }).limit(20)

// Requires compound index: { author: 1, created_at: -1 }

Problem 5: Unindexed Join-Like Queries

// Before: $lookup without an index on the foreign field
// The 'from' collection gets scanned for every document in the pipeline
db.orders.aggregate([
  { $match: { status: 'pending' } },
  { $lookup: {
      from: 'customers',
      localField: 'customer_id',
      foreignField: '_id',
      as: 'customer'
  }}
])

// Fix: Ensure the foreign field has an index
db.customers.createIndex({ _id: 1 })  // Usually exists by default
// For non-_id lookups, index the join field explicitly:
db.orders.createIndex({ customer_id: 1, status: 1 })

Aggregation Pipeline Optimization

Aggregation pipelines introduce unique performance considerations because stages execute sequentially. The order of stages dramatically affects how many documents flow through each pipe segment.

The Cardinal Rule: Filter Early, Project Late

// Suboptimal: $group runs on all documents, then $match filters
db.sales.aggregate([
  { $group: {
      _id: '$region',
      total: { $sum: '$amount' }
  }},
  { $match: { total: { $gte: 10000 } } }  // Filtering AFTER aggregation
])

// Optimal: $match first to reduce the working set, then $group
db.sales.aggregate([
  { $match: { date: { $gte: new Date('2024-01-01') } } },  // Filter EARLY
  { $group: {
      _id: '$region',
      total: { $sum: '$amount' }
  }},
  { $match: { total: { $gte: 10000 } } }  // Then filter aggregates
])

// The first $match benefits from an index on {date: 1}
// and can reduce 10M documents to 500K before grouping

Using explain() on Aggregation Pipelines

// Analyze pipeline execution with executionStats
db.sales.explain('executionStats').aggregate([
  { $match: { region: 'EU', status: 'completed' } },
  { $sort: { created_at: -1 } },
  { $limit: 100 },
  { $project: { _id: 0, amount: 1, customer: 1 } }
])

// The output shows each stage's execution metrics:
// - indexFiltered: documents that passed the $match
// - totalDocsExamined vs nReturned per stage
// - whether $sort spilled to disk (look for 'spilled' in sort stage)

Index-Aware Pipeline Stages

Certain pipeline stages can leverage indexes when placed correctly:

Preventing In-Memory Sort Spills to Disk

// A sort stage exceeding the 100MB memory limit spills to disk (slow)
// Check explain output for: "spilled": true, "spilledBytes": ...

// Fix 1: Add an index that provides the sort order
db.events.createIndex({ timestamp: -1 })

// Fix 2: Increase the memory limit for this operation (use with caution)
db.events.aggregate(
  [
    { $sort: { timestamp: -1 } },
    { $group: { _id: '$type', count: { $sum: 1 } } }
  ],
  { allowDiskUse: true }  // Allows >100MB but slower than indexed sort
)

// Fix 3: Place $match before $sort to reduce the input set
db.events.aggregate([
  { $match: { type: { $in: ['click', 'view'] } } },  // Reduces documents
  { $sort: { timestamp: -1 } }
])

Index Strategies for Query Optimization

ESR Rule: Equality, Sort, Range

When building compound indexes, order your index fields following the ESR guideline:

// Query pattern:
db.orders.find({
  customer_id: 'CUST-123',     // Equality
  status: 'completed',          // Equality
  created_at: { $gte: startDate, $lt: endDate }  // Range
}).sort({ total_amount: -1 })   // Sort

// Optimal ESR index:
db.orders.createIndex({
  customer_id: 1,       // E — Equality
  status: 1,            // E — Equality  
  total_amount: -1,     // S — Sort
  created_at: 1         // R — Range (last)
})

Covered Queries (Index-Only Scans)

A covered query is the holy grail of MongoDB performance — the query can be satisfied entirely from the index without ever touching the documents on disk:

// Create an index that covers all queried and projected fields
db.users.createIndex({
  email: 1,
  status: 1,
  last_login: 1,
  signup_date: 1
})

// This query is now covered — MongoDB reads only from the index
db.users.find(
  { email: 'jane@example.com', status: 'active' },
  { _id: 0, email: 1, last_login: 1, signup_date: 1 }
).explain('executionStats')

// Look for: "totalDocsExamined": 0  — confirms a covered query
// The index contains all fields needed for both filter and projection

Partial Indexes for Selective Filtering

// Index only documents where status is 'active' (much smaller index)
db.users.createIndex(
  { email: 1 },
  { partialFilterExpression: { status: 'active' } }
)

// This query can use the partial index
db.users.find({ email: 'jane@example.com', status: 'active' })

// This query CANNOT use the partial index (status != 'active')
db.users.find({ email: 'jane@example.com' })
// MongoDB knows the index doesn't contain non-active documents

Real-Time Monitoring Commands

Beyond the profiler, MongoDB provides live introspection commands that help you understand current server activity without waiting for profile entries:

Current Operations (Killing Long-Running Queries)

// Find all currently running operations that have been active > 5 seconds
db.currentOp({
  active: true,
  'secs_running': { $gte: 5 }
}).pretty()

// The output includes:
// - opid: operation ID (usable with db.killOp())
// - secs_running: how long it's been executing
// - query: the actual query document
// - planSummary: COLLSCAN or IXSCAN

// Kill a runaway operation by its opid
db.killOp(12345)

Server Status Metrics

// Extract key performance counters from serverStatus
const status = db.serverStatus();

// Documents scanned vs. returned ratio (high ratio = index problems)
console.log('Docs scanned:', status.metrics.document.totalScanned);
console.log('Docs returned:', status.metrics.document.totalReturned);

// Index hits vs. misses
console.log('Index hits:', status.metrics.operation.idxHits);
console.log('Index misses:', status.metrics.operation.idxMisses);

// Query planner cache stats
console.log('Query plan cache size:', status.metrics.queryPlanCache.totalSizeEstimateBytes);
console.log('Plan cache evictions:', status.metrics.queryPlanCache.evicted);

Top Command for Collection-Level Hotspots

// Identify which collections are experiencing the most read/write activity
db.adminCommand('top')
// Returns per-collection time spent in read, write, and other operations
// Useful for spotting unexpectedly hot collections

Production Profiling Best Practices

Driver-Level Performance Monitoring (Node.js Example)

const { MongoClient } = require('mongodb');

const client = new MongoClient('mongodb://localhost:27017', {
  monitorCommands: true  // Enable command monitoring
});

// Subscribe to command events for client-side timing
client.on('commandStarted', (event) => {
  if (event.commandName === 'find' || event.commandName === 'aggregate') {
    console.log(`[APM] Started ${event.commandName} on ${event.databaseName}`,
      'at', new Date(event.time).toISOString());
  }
});

client.on('commandSucceeded', (event) => {
  const duration = event.duration;  // Includes network round-trip time
  if (duration > 100) {
    console.warn(`[APM] Slow query: ${event.commandName} took ${duration}ms`,
      'on', event.databaseName);
  }
});

client.on('commandFailed', (event) => {
  console.error(`[APM] Query failed: ${event.commandName}`,
    'error:', event.failure.message);
});

async function run() {
  await client.connect();
  const db = client.db('mydb');
  
  // This slow query will be captured by both APM and the server profiler
  await db.collection('users').find({
    $where: 'sleep(200) || true'
  }).toArray();
  
  await client.close();
}

run();

Query Plan Cache and Index Hints

MongoDB caches query plans to avoid repeated planning overhead. Sometimes, the cached plan becomes suboptimal as data distribution changes. You can inspect and manipulate the plan cache:

// View all cached plans for a specific query shape on the users collection
db.users.getPlanCache().list()

// Clear the plan cache for the users collection (forces replanning)
db.users.getPlanCache().clear()

// Force MongoDB to use a specific index with a hint
db.users.find({ email: 'jane@example.com', status: 'active' })
  .hint({ email: 1, status: 1 })
  .explain('executionStats')

// Force a collection scan (useful for testing index impact)
db.users.find({ email: 'jane@example.com' })
  .hint({ $natural: 1 })
  .explain('executionStats')

Building a Profiling Dashboard Script

// Complete script: Analyze profiler data and print a summary report
function generateProfileReport(databaseName, thresholdMs = 100) {
  const db = db.getSiblingDB(databaseName);
  
  // 1. Count operations by type in the last hour
  const oneHourAgo = new Date(Date.now() - 3600000);
  
  const opCounts = db.system.profile.aggregate([
    { $match: { ts: { $gte: oneHourAgo }, millis: { $gte: thresholdMs } } },
    { $group: { _id: '$op', count: { $sum: 1 }, avgTime: { $avg: '$millis' } } },
    { $sort: { count: -1 } }
  ]).toArray();
  
  print('\n=== OPERATION BREAKDOWN (Last Hour, >=${thresholdMs}ms) ===');
  opCounts.forEach(op => {
    print(`${op._id.padEnd(10)} | Count: ${op.count.toString().padStart(6)} | Avg ms: ${op.avgTime.toFixed(1)}`);
  });
  
  // 2. Top 5 slowest queries
  const slowest = db.system.profile.find({
    ts: { $gte: oneHourAgo },
    millis: { $gte: thresholdMs },
    op: { $in: ['query', 'command'] }
  }).sort({ millis: -1 }).limit(5).toArray();
  
  print('\n=== TOP 5 SLOWEST QUERIES ===');
  slowest.forEach((entry, i) => {
    print(`#${i + 1} | ${entry.ns} | ${entry.millis}ms | ${entry.planSummary}`);
    print(`   Filter: ${JSON.stringify(entry.command.filter || entry.command.pipeline?.[0]?.$match)}`);
  });
  
  // 3. Collection scans count
  const scans = db.system.profile.count({
    ts: { $gte: oneHourAgo },
    planSummary: { $regex: /COLLSCAN/ }
  });
  
  print(`\n=== COLLECTION SCANS in last hour: ${scans} ===`);
  
  // 4. Worst collections by total time consumed
  const hotCollections = db.system.profile.aggregate([
    { $match: { ts: { $gte: oneHourAgo }, millis: { $gte: thresholdMs } } },
    { $group: {
        _id: '$ns',
        totalMs: { $sum: '$millis' },
        queries: { $sum: 1 },
        p95Ms: { $max: '$millis' }
    }},
    { $sort: { totalMs: -1 } },
    { $limit: 5 }
  ]).toArray();
  
  print('\n=== HOTTEST COLLECTIONS ===');
  hotCollections.forEach(col => {
    print(`${col._id.padEnd(30)} | Total: ${(col.totalMs / 1000).toFixed(1)}s | Queries: ${col.queries} | Worst: ${col.p95Ms}ms`);
  });
  
  print('\n=== REPORT COMPLETE ===');
}

// Run the report
generateProfileReport('mydb', 100);

Conclusion

MongoDB performance profiling is not a one-time activity — it's a continuous feedback loop that should be woven into your development and operations workflows. The profiler gives you the raw signal; explain() gives you the diagnostic detail; and the combination of smart indexing, pipeline restructuring, and vigilant monitoring turns that insight into measurable performance gains. Start with a 100ms threshold on your development and staging environments, run the analysis queries shown above regularly, and treat every collection scan as a bug to fix. The investment in profiling discipline pays for itself many times over as your data grows and your application scales to serve more users with consistent, predictable response times.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles