Introduction to Azure Event Grid Troubleshooting
Azure Event Grid is a fully managed event routing service that enables you to react to state changes in Azure services, third-party services, or your own applications. While Event Grid is designed for high reliability and scalability, developers frequently encounter issues related to event delivery, authentication, filtering, and dead-lettering. This tutorial walks you through the most common Event Grid problems and provides practical solutions with code examples.
Why Troubleshooting Event Grid Matters
Event-driven architectures depend on reliable message delivery. When Event Grid fails to deliver events, downstream systems may miss critical state changes, leading to data inconsistency, broken workflows, and poor user experiences. Understanding how to diagnose and fix these issues quickly is essential for maintaining robust cloud applications. Common pain points include:
- Events not reaching subscribers
- Authentication and webhook validation failures
- Dead-letter queue accumulation
- Event filtering not working as expected
- Throttling and delivery latency
1. Events Not Being Delivered to Subscribers
The most frequent issue developers face is events not arriving at the subscriber endpoint. This can happen for several reasons, including incorrect endpoint configuration, webhook validation handshake failures, or the subscriber returning non-2xx HTTP status codes.
Diagnosing Delivery Failures
Start by checking the delivery metrics in the Azure portal. Navigate to your Event Grid Topic, then select Metrics and look at "Matched Events," "Delivery Attempts," and "Delivery Failed" counters. If matched events are zero, your subscription filter may be excluding everything. If delivery attempts show failures, the subscriber endpoint is likely rejecting the events.
You can also enable diagnostic logs to get detailed delivery information:
# Enable diagnostic settings using Azure CLI
az monitor diagnostic-settings create \
--resource "/subscriptions/{subscription-id}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topic-name}" \
--name "EventGridDiagnostics" \
--logs '[{"category":"DeliveryFailures","enabled":true},{"category":"PublishFailures","enabled":true}]' \
--workspace "/subscriptions/{subscription-id}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace-name}"
Common Cause: Webhook Endpoint Validation
When you create a webhook-based event subscription, Event Grid sends a validation handshake request. If your endpoint does not respond correctly, the subscription creation fails. The validation request includes a validationCode in the body that must be echoed back.
// ASP.NET Core webhook endpoint with validation handshake
[HttpPost]
[Route("api/events")]
public async Task ReceiveEvent()
{
using var reader = new StreamReader(Request.Body);
var body = await reader.ReadToEndAsync();
var events = JsonSerializer.Deserialize<JsonElement[]>(body);
// Handle validation handshake
if (events.Length == 1 && events[0].TryGetProperty("validationCode", out var validationCode))
{
var response = new
{
validationResponse = validationCode.GetString()
};
return Ok(response);
}
// Process actual events
foreach (var evt in events)
{
var eventType = evt.GetProperty("eventType").GetString();
var subject = evt.GetProperty("subject").GetString();
Console.WriteLine($"Received event: {eventType} for {subject}");
}
return Ok();
}
A common mistake is returning a 200 OK without including the validationResponse property. Event Grid requires the exact validation code to be returned in the response body. Alternatively, you can use validation URL handshake by returning a 200 OK and then performing a GET request to the validationUrl provided in the request.
Common Cause: Subscriber Returning Non-2xx Status Codes
Event Grid considers any HTTP response outside the 200-299 range as a delivery failure. If your subscriber returns a 500 error due to an unhandled exception, Event Grid will retry with exponential backoff. After the retry limit is exhausted, the event is sent to the dead-letter queue (if configured) or dropped.
// Node.js Express endpoint with proper error handling
const express = require('express');
const app = express();
app.use(express.json());
app.post('/api/events', async (req, res) => {
const events = req.body;
for (const event of events) {
try {
// Check for validation handshake
if (event.validationCode) {
return res.status(200).json({
validationResponse: event.validationCode
});
}
// Process the event
await processEvent(event);
} catch (error) {
console.error(`Failed to process event ${event.id}:`, error);
// Return 400 for bad data (no retry) or 500 for transient (retry)
if (error.isPermanent) {
return res.status(400).json({ error: 'Bad request' });
}
return res.status(500).json({ error: 'Internal error' });
}
}
res.status(200).json({ status: 'processed' });
});
function processEvent(event) {
return new Promise((resolve, reject) => {
// Your event processing logic here
console.log(`Processing event: ${event.eventType}`);
resolve();
});
}
app.listen(3000, () => console.log('Webhook listening on port 3000'));
2. Configuring and Using Dead-Letter Queues
Without a dead-letter queue (DLQ), events that exhaust all retry attempts are permanently lost. Configuring a DLQ ensures that failed events are stored in a storage account blob for later inspection and reprocessing.
Setting Up Dead-Letter Storage
To configure a DLQ, you need a Storage Account and a blob container. You then reference it when creating or updating the event subscription.
# Create a storage account for dead-lettering
az storage account create \
--name "egdlqstorage" \
--resource-group "myResourceGroup" \
--location "eastus" \
--sku "Standard_LRS"
# Get the storage account ID
STORAGE_ID=$(az storage account show \
--name "egdlqstorage" \
--resource-group "myResourceGroup" \
--query id -o tsv)
# Create an event subscription with dead-letter and retry policies
az eventgrid event-subscription create \
--name "myEventSubscription" \
--source-resource-id "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.EventGrid/topics/myTopic" \
--endpoint "https://myapp.azurewebsites.net/api/events" \
--deadletter-endpoint "$STORAGE_ID/blobServices/default/containers/eventgrid-deadletter" \
--max-delivery-attempts 10 \
--event-ttl 1440
Inspecting Dead-Lettered Events
Dead-lettered events are stored as JSON blobs in the specified container. The blob name follows a pattern that includes the topic name, subscription name, timestamp, and a unique identifier. You can retrieve and reprocess them programmatically.
// C# - Retrieve and reprocess dead-lettered events
using Azure.Storage.Blobs;
using System.Text.Json;
var blobServiceClient = new BlobServiceClient("DefaultEndpointsProtocol=https;AccountName=egdlqstorage;AccountKey=...;EndpointSuffix=core.windows.net");
var containerClient = blobServiceClient.GetBlobContainerClient("eventgrid-deadletter");
await foreach (var blobItem in containerClient.GetBlobsAsync())
{
var blobClient = containerClient.GetBlobClient(blobItem.Name);
var downloadResult = await blobClient.DownloadContentAsync();
var eventJson = downloadResult.Value.Content.ToString();
// Parse the dead-lettered event
using var doc = JsonDocument.Parse(eventJson);
var root = doc.RootElement;
// The dead-letter blob contains the original event plus delivery metadata
if (root.TryGetProperty("deliveryAttempts", out var attempts))
{
Console.WriteLine($"Event {root.GetProperty("id").GetString()} failed after {attempts.GetArrayLength()} attempts");
}
// Reprocess the event
var originalEvent = root.GetProperty("event").GetRawText();
await ReprocessEvent(originalEvent);
// Optionally delete the blob after successful reprocessing
await blobClient.DeleteAsync();
}
async Task ReprocessEvent(string eventJson)
{
// Send the event back to the topic or process it directly
Console.WriteLine($"Reprocessing event: {eventJson}");
}
3. Event Filtering Issues
Event Grid supports advanced filtering, but misconfigured filters are a common source of problems. If subscribers are not receiving events, the filter may be too restrictive or referencing the wrong property.
Understanding Filter Types
Event Grid offers three types of filters: subject filters, event type filters, and advanced filters. A common mistake is confusing the subject property with custom data properties. The subject is a top-level field in the event schema, while advanced filters can target any property in the data object.
# Create an event subscription with advanced filtering
az eventgrid event-subscription create \
--name "filteredSubscription" \
--source-resource-id "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.EventGrid/topics/myTopic" \
--endpoint "https://myapp.azurewebsites.net/api/events" \
--subject-begins-with "/orders/" \
--included-event-types "order.created" "order.updated" \
--advanced-filter data.amount NumberGreaterThanOrEquals 100 \
--advanced-filter data.region StringIn "eastus" "westus"
Debugging Filter Mismatches
If your filter is not matching events as expected, publish a test event and check the "Matched Events" metric. A value of zero means the filter is excluding the event. Here is how to publish a test event and verify the payload structure:
# Publish a test event using Azure CLI
az eventgrid event publish \
--endpoint "https://mytopic.eastus-1.eventgrid.azure.net/api/events" \
--key "your-topic-key" \
--id "$(uuidgen)" \
--subject "/orders/12345" \
--event-type "order.created" \
--data '{
"orderId": "12345",
"amount": 150.00,
"region": "eastus",
"customer": "Contoso Ltd"
}'
Verify that the subject in your event matches the --subject-begins-with filter. Also ensure that the data properties referenced in advanced filters exist and have the correct data types. A NumberGreaterThanOrEquals filter on a string value will never match.
Using ARM Template for Complex Filters
For complex filtering scenarios, ARM templates provide better control and versioning. The following template demonstrates a subscription with multiple advanced filters:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"topicName": { "type": "string" },
"subscriptionName": { "type": "string" },
"endpointUrl": { "type": "string" }
},
"resources": [
{
"type": "Microsoft.EventGrid/eventSubscriptions",
"apiVersion": "2021-12-01",
"name": "[concat(parameters('topicName'), '/', parameters('subscriptionName'))]",
"properties": {
"destination": {
"endpointType": "WebHook",
"properties": {
"endpointUrl": "[parameters('endpointUrl')]"
}
},
"filter": {
"subjectBeginsWith": "/orders/",
"includedEventTypes": [ "order.created", "order.updated" ],
"advancedFilters": [
{
"operatorType": "NumberGreaterThanOrEquals",
"key": "data.amount",
"value": 100
},
{
"operatorType": "StringIn",
"key": "data.region",
"values": [ "eastus", "westus" ]
}
]
},
"retryPolicy": {
"maxDeliveryAttempts": 10,
"eventTimeToLiveInMinutes": 1440
}
}
}
]
}
4. Authentication and Security Issues
Event Grid supports multiple authentication mechanisms for webhook endpoints, including key-based validation, Microsoft Entra ID (formerly Azure AD), and managed identities. Authentication failures are common when the subscriber does not implement the expected validation mechanism.
Webhook Endpoint Validation with AAD
When using Microsoft Entra ID for webhook authentication, Event Grid includes a bearer token in the Authorization header. The subscriber must validate this token against the Event Grid resource. A common mistake is not configuring the correct application ID or not validating the token audience.
// C# - Validate AAD token from Event Grid webhook
using Microsoft.IdentityModel.Tokens;
using Microsoft.IdentityModel.JsonWebTokens;
using System.IdentityModel.Tokens.Jwt;
[HttpPost]
[Route("api/events")]
public async Task<IActionResult> ReceiveEvent()
{
// Handle validation handshake first (no auth header for validation)
using var reader = new StreamReader(Request.Body);
var body = await reader.ReadToEndAsync();
var events = JsonSerializer.Deserialize<JsonElement[]>(body);
if (events.Length == 1 && events[0].TryGetProperty("validationCode", out var code))
{
return Ok(new { validationResponse = code.GetString() });
}
// Validate the AAD token
var authHeader = Request.Headers["Authorization"].FirstOrDefault();
if (string.IsNullOrEmpty(authHeader) || !authHeader.StartsWith("Bearer "))
{
return Unauthorized();
}
var token = authHeader.Substring("Bearer ".Length).Trim();
var tokenHandler = new JwtSecurityTokenHandler();
// Event Grid resource application ID
var eventGridAppId = "4962773b-9cdb-44cf-a8bf-237846a00ab7";
var validationParameters = new TokenValidationParameters
{
ValidAudience = eventGridAppId,
ValidIssuer = $"https://login.microsoftonline.com/{tenantId}/v2.0",
IssuerSigningKeys = await GetSigningKeys(),
ValidateLifetime = true
};
try
{
var principal = tokenHandler.ValidateToken(token, validationParameters, out _);
// Token is valid, process events
return Ok();
}
catch (SecurityTokenException ex)
{
Console.WriteLine($"Token validation failed: {ex.Message}");
return Unauthorized();
}
}
Managed Identity for Event Subscription
When delivering events to services like Service Bus, Event Hubs, or Storage Queues, you can use a managed identity instead of connection strings. This eliminates the need to manage secrets but requires proper role assignments.
# Assign the Event Grid Topic's managed identity to the Service Bus Sender role
az role assignment create \
--assignee "<topic-managed-identity-object-id>" \
--role "Azure Service Bus Data Sender" \
--scope "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.ServiceBus/namespaces/myServiceBusNamespace"
# Create event subscription using managed identity
az eventgrid event-subscription create \
--name "serviceBusSubscription" \
--source-resource-id "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.EventGrid/topics/myTopic" \
--endpoint-type "servicebusqueue" \
--endpoint "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.ServiceBus/namespaces/myServiceBusNamespace/queues/myqueue" \
--delivery-identity "systemassigned"
5. Throttling and Performance Issues
Event Grid has throughput limits that vary by topic type and region. Custom topics have a default limit of 5,000 events per second per region. If you exceed this limit, Event Grid returns HTTP 429 (Too Many Requests) responses.
Handling Throttling on Publish
When publishing events, implement retry logic with exponential backoff to handle 429 responses. The EventGridPublisherClient in the Azure SDK handles retries automatically, but you should configure the retry policy appropriately.
// C# - Configure retry policy for Event Grid publisher
using Azure.Messaging.EventGrid;
using Azure.Core;
var credential = new AzureKeyCredential("your-topic-key");
var options = new EventGridPublisherClientOptions
{
Retry =
{
MaxRetries = 5,
Delay = TimeSpan.FromSeconds(1),
MaxDelay = TimeSpan.FromSeconds(30),
Mode = RetryMode.Exponential
}
};
var client = new EventGridPublisherClient(
new Uri("https://mytopic.eastus-1.eventgrid.azure.net/api/events"),
credential,
options
);
// Batch publish events for better throughput
var events = new List<EventGridEvent>();
for (int i = 0; i < 1000; i++)
{
events.Add(new EventGridEvent(
subject: $"/orders/{i}",
eventType: "order.created",
dataVersion: "1.0",
data: new BinaryData($"{{\"orderId\":\"{i}\",\"amount\":{i * 10}}}"))
);
}
try
{
await client.SendEventsAsync(events);
Console.WriteLine($"Successfully published {events.Count} events");
}
catch (RequestFailedException ex) when (ex.Status == 429)
{
Console.WriteLine("Throttled. Consider reducing publish rate or batching.");
// Implement custom backoff or queue events for later delivery
}
Subscriber-Side Throttling
If your subscriber cannot process events fast enough, Event Grid will queue events and retry. However, sustained slow processing can lead to event expiration. Consider using the Service Bus or Storage Queue as an intermediary to buffer events.
// Python - Batch process events from Event Grid via Service Bus
from azure.servicebus import ServiceBusClient, ServiceBusMessage
from azure.servicebus.aio import ServiceBusClient as AsyncServiceBusClient
import asyncio
import json
CONNECTION_STR = "Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=..."
QUEUE_NAME = "eventgrid-queue"
async def process_events_batch():
async with AsyncServiceBusClient.from_connection_string(CONNECTION_STR) as client:
async with client.get_queue_receiver(QUEUE_NAME) as receiver:
# Receive a batch of up to 100 messages
messages = await receiver.receive_messages(max_message_count=100, max_wait_time=5)
if not messages:
print("No messages to process")
return
for message in messages:
try:
event_data = json.loads(str(message))
await handle_event(event_data)
await receiver.complete_message(message)
except Exception as e:
print(f"Failed to process message: {e}")
# Abandon the message for retry
await receiver.abandon_message(message)
print(f"Processed {len(messages)} events")
async def handle_event(event):
# Your event handling logic
print(f"Processing event: {event.get('eventType')}")
# Run the processor
asyncio.run(process_events_batch())
6. CloudEvents Schema Compatibility
Event Grid supports the CloudEvents 1.0 specification, but switching between Event Grid schema and CloudEvents schema can cause issues if subscribers are not updated to handle the new format. The property names and structure differ between schemas.
Comparing Event Schemas
In the Event Grid schema, properties use camelCase (eventType, eventTime), while CloudEvents uses lowercase (type, time). If your subscriber expects one schema but receives another, deserialization will fail.
// Event Grid Schema
{
"id": "1234-5678",
"eventType": "order.created",
"subject": "/orders/12345",
"eventTime": "2024-01-15T10:30:00Z",
"data": { "orderId": "12345", "amount": 150.00 },
"dataVersion": "1.0",
"topic": "/subscriptions/.../topics/myTopic"
}
// CloudEvents 1.0 Schema
{
"id": "1234-5678",
"type": "order.created",
"source": "/orders/12345",
"time": "2024-01-15T10:30:00Z",
"data": { "orderId": "12345", "amount": 150.00 },
"specversion": "1.0",
"subject": "/orders/12345"
}
Configuring Input Schema
When creating a custom topic, you can specify the input schema. Once set, all published events must conform to that schema. You cannot change the input schema after topic creation.
# Create a topic with CloudEvents schema
az eventgrid topic create \
--name "myCloudEventsTopic" \
--resource-group "myResourceGroup" \
--location "eastus" \
--input-schema "cloudeventschemav1_0"
# Publish an event in CloudEvents format
az eventgrid event publish \
--endpoint "https://myCloudEventsTopic.eastus-1.eventgrid.azure.net/api/events" \
--key "your-topic-key" \
--id "$(uuidgen)" \
--source "/orders/12345" \
--type "order.created" \
--data '{
"orderId": "12345",
"amount": 150.00
}'
7. Best Practices for Event Grid Reliability
Always Configure Dead-Letter Queues
Never create an event subscription without a dead-letter queue. Without it, events that fail delivery after all retries are permanently lost. The storage cost for dead-lettered events is minimal, and having them available for reprocessing can save you from data loss scenarios.
Implement Idempotent Event Handlers
Event Grid provides "at least once" delivery guarantees, meaning events may be delivered more than once. Your subscriber must handle duplicate events gracefully by implementing idempotency checks.
// C# - Idempotent event handler using event ID tracking
public class OrderEventHandler
{
private readonly ICacheService _cache;
public async Task HandleAsync(EventGridEvent eventGridEvent)
{
// Check if this event was already processed
var processedKey = $"processed:{eventGridEvent.Id}";
if (await _cache.ExistsAsync(processedKey))
{
Console.WriteLine($"Event {eventGridEvent.Id} already processed. Skipping.");
return;
}
// Process the event
var orderData = eventGridEvent.Data.ToObjectFromJson<OrderData>();
await CreateOrderAsync(orderData);
// Mark as processed with TTL to eventually clean up
await _cache.SetAsync(processedKey, "1", TimeSpan.FromHours(24));
}
private async Task CreateOrderAsync(OrderData order)
{
// Order creation logic
Console.WriteLine($"Creating order {order.OrderId}");
}
}
public class OrderData
{
public string OrderId { get; set; }
public decimal Amount { get; set; }
}
Use Batch Event Publishing
Publishing events one at a time is inefficient and increases the chance of hitting throttling limits. Batch publishing allows you to send up to 1 MB of events in a single request, significantly improving throughput.
Monitor with Alerts
Set up Azure Monitor alerts for critical Event Grid metrics, including delivery failures, matched events dropping to zero, and dead-letter queue growth. Proactive monitoring helps you catch issues before they impact your application.
# Create an alert for delivery failures
az monitor metrics alert create \
--name "EventGridDeliveryFailures" \
--resource-group "myResourceGroup" \
--scopes "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/Microsoft.EventGrid/topics/myTopic" \
--condition "avg DeliveryFailed > 0" \
--window-size 5m \
--evaluation-frequency 1m \
--action "/subscriptions/{sub-id}/resourceGroups/myResourceGroup/providers/microsoft.insights/actionGroups/myActionGroup"
Choose the Right Endpoint Type
Webhooks are simple but require your endpoint to be publicly accessible and respond within 30 seconds. For internal services or long-running processing, consider using Service Bus Queues, Storage Queues, or Event Hubs as delivery endpoints. These provide built-in buffering and decouple event delivery from processing.
Conclusion
Troubleshooting Azure Event Grid requires a systematic approach: verify event delivery metrics, check webhook validation handshakes, inspect dead-letter queues, validate filter configurations, and monitor for throttling. By implementing the solutions and best practices covered in this tutorial—configuring dead-letter queues, building idempotent handlers, using batch publishing, and setting up proactive alerts—you can build resilient event-driven architectures that gracefully handle failures and maintain reliable event delivery across your cloud applications.