Troubleshooting CosmosDB: Common Issues and Solutions

Introduction to Troubleshooting CosmosDB

Azure CosmosDB is a globally distributed, multi-model database service designed for low-latency, high-throughput applications. While it offers impressive scalability and performance, developers frequently encounter issues related to throttling, partitioning, consistency, and query performance. This tutorial walks through the most common CosmosDB problems and provides practical, code-driven solutions to diagnose and resolve them.

Why Troubleshooting CosmosDB Matters

Unlike traditional relational databases, CosmosDB operates on a request-unit (RU) consumption model and relies heavily on proper partition key design. Small misconfigurations can lead to silent performance degradation, unexpected billing spikes, or even application outages. Understanding how to identify and fix these issues is critical for maintaining reliable, cost-effective cloud applications.

Issue 1: Request Rate Too Large (429 Throttling)

The most common CosmosDB error is HTTP 429, which occurs when your application consumes more Request Units (RUs) per second than your provisioned throughput. The SDK typically retries automatically, but excessive throttling degrades latency and reliability.

Diagnosing Throttling

Monitor the RequestCharge and StatusCode properties on every response. The .NET SDK exposes these through Response objects.

using Microsoft.Azure.Cosmos;

var cosmosClient = new CosmosClient(connectionString);
var container = cosmosClient.GetContainer("MyDatabase", "MyContainer");

try
{
    ItemResponse<MyDocument> response = await container.ReadItemAsync<MyDocument>(
        "doc-123",
        new PartitionKey("partition-A"));

    Console.WriteLine($"Request Charge: {response.RequestCharge} RUs");
    Console.WriteLine($"Status: {response.StatusCode}");
}
catch (CosmosException ex) when (ex.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
{
    Console.WriteLine($"Throttled! Retry after: {ex.RetryAfter} ms");
    Console.WriteLine($"Activity ID: {ex.ActivityId}");
}

Solutions for Throttling

Increase provisioned throughput manually or enable autoscale to handle bursty workloads.
Optimize queries to consume fewer RUs by adding composite indexes or filtering on the partition key.
Implement exponential backoff with jitter when the SDK's built-in retries are insufficient.
Batch operations using TransactionalBatch to reduce round-trips and RU overhead.

// Example: Batching multiple writes to reduce RU consumption
TransactionalBatch batch = container.CreateTransactionalBatch(new PartitionKey("partition-A"));
batch.CreateItem(new MyDocument { Id = "doc-1", Name = "Alpha" });
batch.CreateItem(new MyDocument { Id = "doc-2", Name = "Beta" });
batch.CreateItem(new MyDocument { Id = "doc-3", Name = "Gamma" });

TransactionalBatchResponse batchResponse = await batch.ExecuteAsyncAsync();
Console.WriteLine($"Batch consumed {batchResponse.RequestCharge} RUs total");

Issue 2: Hot Partitions

A hot partition occurs when a disproportionate amount of traffic targets a single logical partition. Because CosmosDB distributes throughput evenly across physical partitions, a hot partition becomes a bottleneck even when overall provisioned RUs are high.

Identifying Hot Partitions

Use Azure Monitor metrics to inspect Normalized RU Consumption by partition key range. If one range consistently hits 100% while others remain low, you have a hot partition. You can also log partition keys in your application to detect skew.

// Log partition key usage to detect skew
ItemResponse<MyDocument> response = await container.CreateItemAsync(document, new PartitionKey(document.Category));

var telemetry = new
{
    PartitionKey = document.Category,
    RequestCharge = response.RequestCharge,
    Timestamp = DateTime.UtcNow
};
logger.LogInformation("CosmosDB Write: {@Telemetry}", telemetry);

Resolving Hot Partitions

Choose a high-cardinality partition key that distributes writes evenly, such as a user ID or device ID rather than a status field.
Avoid synthetic hot keys like "latest" or "active" that all clients target simultaneously.
Use a synthetic partition key by concatenating fields (e.g., userId + "-" + categoryId) to increase cardinality.
Consider subpartitioning for hierarchical data patterns.

Issue 3: Cross-Partition Queries

Queries that do not include the partition key trigger a cross-partition scan, consuming significantly more RUs and increasing latency. This is a frequent cause of unexpectedly high costs.

Detecting Cross-Partition Queries

The SDK exposes diagnostic information that reveals whether a query fanned out across partitions.

var query = container.GetItemQueryIterator<MyDocument>(
    "SELECT * FROM c WHERE c.Status = 'Active'");

while (query.HasMoreResults)
{
    FeedResponse<MyDocument> page = await query.ReadNextAsync();
    Console.WriteLine($"Retrieved {page.Count} documents");
    Console.WriteLine($"Request Charge: {page.RequestCharge} RUs");
    Console.WriteLine($"Index Metrics: {page.IndexMetrics}");
}

If RequestCharge is high relative to the number of documents returned, and IndexMetrics shows full index scans, your query is likely cross-partition.

Optimizing Queries

Always include the partition key in your WHERE clause when possible.
Add composite indexes for multi-property filters.
Use TOP or LIMIT to cap RU consumption during development and testing.
Avoid SELECT * and project only the properties you need.

// Optimized query with partition key and composite index
var optimizedQuery = container.GetItemQueryIterator<MyDocument>(
    new QueryDefinition("SELECT c.Id, c.Name FROM c WHERE c.PartitionKey = @pk AND c.Status = @status")
        .WithParameter("@pk", "partition-A")
        .WithParameter("@status", "Active"),
    requestOptions: new QueryRequestOptions { PartitionKey = new PartitionKey("partition-A") });

Issue 4: Consistency-Related Anomalies

CosmosDB offers five consistency levels: Strong, Bounded Staleness, Session, Consistent Prefix, and Eventual. Misunderstanding consistency can lead to stale reads or unexpected behavior in multi-region deployments.

Common Consistency Pitfalls

With Session consistency (the default), a single client sees its own writes, but other clients may see stale data. This often surfaces as "I wrote it but can't read it" bugs in load-balanced applications.

// Explicitly request a higher consistency for critical reads
var readOptions = new ItemRequestOptions
{
    ConsistencyLevel = ConsistencyLevel.Strong
};

ItemResponse<MyDocument> response = await container.ReadItemAsync<MyDocument>(
    "doc-123",
    new PartitionKey("partition-A"),
    readOptions);

Best Practices for Consistency

Use Session consistency for most workloads; it balances performance and correctness.
Upgrade to Strong only for specific critical reads, not as a database-wide default.
Leverage ETag and optimistic concurrency to handle concurrent updates safely.
Design for idempotency so eventual consistency does not corrupt state.

// Optimistic concurrency using ETag
var doc = await container.ReadItemAsync<MyDocument>("doc-123", new PartitionKey("partition-A"));
doc.Resource.Status = "Updated";

try
{
    await container.ReplaceItemAsync(
        doc.Resource,
        "doc-123",
        new PartitionKey("partition-A"),
        new ItemRequestOptions { IfMatchEtag = doc.ETag });
}
catch (CosmosException ex) when (ex.StatusCode == System.Net.HttpStatusCode.PreconditionFailed)
{
    Console.WriteLine("Document was modified by another process. Retry the operation.");
}

Issue 5: Connection and Timeout Errors

Intermittent SocketException, TaskCanceledException, or gateway timeout errors often stem from connection mode configuration or network issues between your application and CosmosDB.

Configuring Direct Mode

Direct mode with TCP is recommended for high-performance workloads because it bypasses the gateway and reduces latency.

var clientOptions = new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Direct,
    Protocol = CosmosProtocol.Tcp,
    RequestTimeout = TimeSpan.FromSeconds(30),
    OpenTcpConnectionTimeout = TimeSpan.FromSeconds(5),
    MaxRequestsPerTcpConnection = 30,
    MaxTcpConnectionsPerEndpoint = 30
};

var cosmosClient = new CosmosClient(connectionString, clientOptions);

Troubleshooting Connectivity

Use Direct mode for latency-sensitive workloads; Gateway mode is simpler but adds a hop.
Adjust connection pool settings for high-concurrency scenarios.
Check network security rules and ensure the CosmosDB endpoint is reachable from your compute environment.
Enable SDK diagnostics to capture detailed timing information for every request.

// Enable detailed diagnostics
var diagnosticsOptions = new CosmosClientOptions
{
    ConnectionMode = ConnectionMode.Direct,
    EnableTcpConnectionEndpointRediscovery = true
};

var client = new CosmosClient(connectionString, diagnosticsOptions);

// Capture diagnostics on a specific operation
ItemResponse<MyDocument> response = await container.ReadItemAsync<MyDocument>(
    "doc-123", new PartitionKey("partition-A"));
Console.WriteLine(response.Diagnostics.ToString());

Best Practices for CosmosDB Reliability

Design partition keys early — changing partition keys requires creating a new container and migrating data.
Monitor RU consumption continuously using Azure Monitor alerts and Application Insights.
Use autoscale throughput for unpredictable workloads to avoid manual scaling and throttling.
Implement proper retry logic beyond the SDK defaults for mission-critical paths.
Profile queries in development using IndexMetrics and RequestCharge before deploying to production.
Store large binaries externally (e.g., Blob Storage) and keep only metadata in CosmosDB to reduce RU costs.
Leverage change feed for event-driven architectures instead of polling queries.

Conclusion

Troubleshooting CosmosDB effectively requires understanding its unique architecture around request units, partitioning, and consistency. By monitoring RU consumption, choosing the right partition keys, optimizing queries to stay within a single partition, configuring connection modes properly, and applying the right consistency level for each workload, you can avoid the vast majority of common issues. The key is proactive instrumentation — always log request charges, activity IDs, and diagnostics so that when problems arise, you have the data needed to resolve them quickly. With these practices in place, CosmosDB can deliver the low-latency, globally distributed performance it was designed for.