Scaling Event Grid: From Prototype to Production

Scaling Event Grid: From Prototype to Production

Azure Event Grid is a fully managed event routing service that enables you to build reactive, event-driven applications with minimal infrastructure overhead. While building a prototype with Event Grid is deceptively simple—create a topic, subscribe, and publish—moving that same architecture to production introduces a host of challenges around throughput, reliability, ordering, cost, and observability. This tutorial walks you through the journey from a working prototype to a hardened, production-grade event-driven system.

What Is Event Grid?

Event Grid is a publish-subscribe service that routes events from sources (such as Azure services, custom applications, or third-party systems) to handlers (such as Azure Functions, Logic Apps, webhooks, or Service Bus queues). Unlike message brokers such as Service Bus or Kafka, Event Grid is optimized for lightweight, discrete event notifications rather than long-lived message streams or heavy payloads.

Event Grid supports two primary resource models:

• Event Grid Topics — Custom endpoints where publishers send events, and subscribers attach filters to receive them.
• Event Grid Domains — A higher-level container that lets you manage thousands of topics under a single endpoint, ideal for multi-tenant scenarios.

Understanding which model fits your workload is the first step in scaling successfully.

Why Scaling Matters

In a prototype, you might publish a few hundred events per minute to a single topic with one subscriber. In production, you may need to handle millions of events per hour, fan out to dozens of subscribers, guarantee delivery under partial failures, and keep costs predictable. Without deliberate design, you will hit bottlenecks such as:

• Throttling on webhook subscribers that cannot keep up with event volume.
• Dead-letter queues filling up due to transient downstream failures.
• Unbounded retry storms degrading publisher latency.
• Cost surprises from per-event pricing at high volumes.
• Difficulty tracing events across a complex fan-out topology.

The good news is that Event Grid is designed to scale horizontally. The challenge is configuring and orchestrating it correctly.

From Prototype to Production: A Practical Walkthrough

Let us start with a typical prototype. You have an Azure Function that publishes order events to a custom topic, and another Function subscribed to that topic that updates inventory. Here is the prototype publisher:

using Azure.Messaging.EventGrid;
using System.Threading.Tasks;

public class OrderPublisher
{
    private readonly EventGridPublisherClient _client;

    public OrderPublisher(string topicEndpoint, string topicKey)
    {
        _client = new EventGridPublisherClient(
            new Uri(topicEndpoint),
            new AzureKeyCredential(topicKey));
    }

    public async Task PublishOrderAsync(Order order)
    {
        var eventGridEvent = new EventGridEvent(
            subject: $"orders/{order.Id}",
            eventType: "order.created",
            dataVersion: "1.0",
            data: new BinaryData(order));

        await _client.SendEventAsync(eventGridEvent);
    }
}

This works fine for a prototype. But in production, several improvements are necessary.

1. Batch Your Publishes

Each call to SendEventAsync incurs network latency and is billed as a single operation. At scale, you should batch events using SendEventsAsync:

public async Task PublishOrdersBatchAsync(IEnumerable<Order> orders)
{
    var events = orders.Select(order => new EventGridEvent(
        subject: $"orders/{order.Id}",
        eventType: "order.created",
        dataVersion: "1.0",
        data: new BinaryData(order))).ToList();

    // Event Grid accepts up to 1 MB per batch and 100,000 events per second per topic
    await _client.SendEventsAsync(events);
}

Batching reduces per-event overhead and improves throughput dramatically. Keep each batch under the 1 MB size limit, and consider chunking large batches accordingly.

2. Use Event Grid Domains for Multi-Tenancy

If you are serving multiple tenants or business units, creating a separate topic for each one becomes operationally expensive. Event Grid Domains let you publish to many topics through a single endpoint using a topic name in the event's channel header:

var eventGridEvent = new EventGridEvent(
    subject: $"tenantA/orders/{order.Id}",
    eventType: "order.created",
    dataVersion: "1.0",
    data: new BinaryData(order));

// The topic name is specified when sending to a domain
await _domainClient.SendEventAsync(eventGridEvent, "tenantA-orders");

Subscribers can then attach to individual topics within the domain, with independent filtering and dead-letter configuration per tenant.

3. Configure Dead Letter and Retry Policies

In production, subscribers will fail. Without a dead-letter destination, failed events are silently dropped after the retry policy is exhausted. Always configure a dead-letter endpoint, typically a Storage Account container:

az eventgrid event-subscription create \
  --name order-processor-sub \
  --source-resource-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topic} \
  --endpoint-type azurefunction \
  --endpoint /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Web/sites/{functionApp}/functions/ProcessOrder \
  --deadletter-endpoint /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{storage}/blobServices/default/containers/eventgrid-deadletter \
  --max-delivery-attempts 10 \
  --event-ttl 1440

This configuration retries delivery up to 10 times with exponential backoff and stores undeliverable events in a blob container for later inspection and replay.

4. Implement Idempotent Subscribers

Event Grid provides at-least-once delivery. This means your subscriber may receive the same event more than once, especially during retries or failover. Your handlers must be idempotent. A common pattern is to track processed event IDs:

public static class ProcessOrder
{
    [FunctionName("ProcessOrder")]
    public static async Task Run(
        [EventGridTrigger] EventGridEvent eventGridEvent,
        [CosmosDB("orders", "processed-events", ConnectionStringSetting = "CosmosDb")] 
            IAsyncCollector<ProcessedEvent> processedStore,
        ILogger log)
    {
        var eventId = eventGridEvent.Id;

        // Check if already processed (idempotency guard)
        if (await AlreadyProcessedAsync(eventId))
        {
            log.LogInformation($"Event {eventId} already processed. Skipping.");
            return;
        }

        var order = eventGridEvent.Data.ToObjectFromJson<Order>();
        await UpdateInventoryAsync(order);

        await processedStore.AddAsync(new ProcessedEvent
        {
            Id = eventId,
            ProcessedAt = DateTime.UtcNow
        });
    }
}

Using Cosmos DB with a unique constraint on the event ID provides a strong idempotency guarantee even under concurrent deliveries.

5. Filter Aggressively at the Subscription Level

One of the most overlooked scaling techniques is filtering. Event Grid supports subject, event type, and advanced attribute filtering. Pushing filtering down to Event Grid means subscribers only receive events they actually care about, reducing load and cost:

az eventgrid event-subscription create \
  --name high-value-orders-sub \
  --source-resource-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topic} \
  --endpoint-type azurefunction \
  --endpoint /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Web/sites/{functionApp}/functions/ProcessHighValueOrder \
  --included-event-types order.created \
  --subject-begins-with "orders/premium/" \
  --advanced-filter data.amount GreaterThan 1000

This subscription only receives premium orders over $1,000. Other events never reach the subscriber, saving compute and delivery costs.

6. Handle Backpressure with Service Bus Integration

Webhook and Azure Function subscribers have concurrency limits. If your event volume spikes, the subscriber may become overwhelmed and start returning 429 or 5xx responses, triggering retries and amplifying load. A robust pattern is to use Event Grid to push events into a Service Bus queue, then have your worker consume from the queue at its own pace:

az eventgrid event-subscription create \
  --name orders-to-servicebus \
  --source-resource-id /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topic} \
  --endpoint-type servicebusqueue \
  --endpoint /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.ServiceBus/namespaces/{namespace}/queues/orders

Service Bus provides buffering, dead-letter queues, sessions for ordering, and competing consumers for horizontal scaling. This decouples Event Grid's delivery speed from your processing speed.

7. Enable Diagnostic Logging and Tracing

In production, you need visibility into delivery failures, latency, and dropped events. Enable diagnostic settings on your Event Grid topic to send logs to Log Analytics:

az monitor diagnostic-settings create \
  --name eventgrid-diagnostics \
  --resource /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.EventGrid/topics/{topic} \
  --workspace /subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{workspace} \
  --logs '[{"category":"DeliveryFailures","enabled":true},{"category":"PublishFailures","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

You can then query delivery failures in Kusto:

EventGridLogs
| where CategoryName == "DeliveryFailures"
| project TimeGenerated, EventId, Subject, EventType, Error
| order by TimeGenerated desc

For end-to-end tracing, propagate a correlation ID in the event data and log it at every hop. This lets you trace a single event from publisher through Event Grid to the final subscriber.

Best Practices Summary

• Batch publishes to reduce latency and per-operation cost.
• Use Event Grid Domains when managing more than a handful of related topics or tenants.
• Always configure dead-letter destinations — never rely on default behavior in production.
• Design subscribers to be idempotent using event ID tracking.
• Filter at the subscription level to reduce unnecessary deliveries.
• Introduce Service Bus or Storage Queues as a buffer when subscribers cannot keep up with event volume.
• Keep payloads small — Event Grid is designed for notifications, not bulk data transfer. Use a claim-check pattern with Storage or Cosmos DB for large payloads.
• Monitor delivery latency and failure rates with diagnostic logs and alerts.
• Use managed identities instead of shared access keys for both publishing and subscribing to improve security and simplify key rotation.
• Version your event schemas using the dataVersion field so subscribers can handle schema evolution gracefully.

Conclusion

Scaling Azure Event Grid from prototype to production is less about raw throughput—which the service handles natively—and more about thoughtful architecture around it. By batching publishes, embracing domains for multi-tenancy, configuring dead-lettering and retries, building idempotent subscribers, filtering aggressively, buffering with Service Bus when needed, and investing in observability, you can build an event-driven system that remains reliable and cost-effective under real-world load. The prototype gets you to a working demo; these production patterns keep you running when traffic, tenants, and complexity grow.