Scaling API Gateway: From Prototype to Production

Scaling API Gateway: From Prototype to Production

An API Gateway is the single entry point that sits between your clients and your backend services. It handles request routing, composition, authentication, rate limiting, observability, and protocol translation. While building a prototype API gateway is straightforward — often just a reverse proxy with a few routes — scaling it to production traffic requires careful planning around performance, resilience, security, and observability.

This tutorial walks you through the journey of taking an API gateway from a simple prototype to a production-ready, horizontally scalable system. We will cover architecture decisions, configuration, rate limiting, caching, autoscaling, and monitoring, with practical code examples along the way.

What Is an API Gateway?

An API gateway acts as a front-door facade for one or more backend services. Instead of clients calling each microservice directly, they call the gateway, which routes requests to the appropriate service. The gateway can also aggregate responses, transform payloads, enforce security policies, and collect metrics.

Popular API gateway solutions include Kong, NGINX, AWS API Gateway, Tyk, Traefik, and Envoy. In this tutorial, we will use a combination of NGINX for configuration examples and Node.js for custom middleware logic, since these are widely understood and easy to adapt.

Why Scaling Matters

A prototype gateway running on a single instance might handle a few hundred requests per second. In production, you may need to handle thousands or tens of thousands of requests per second with low latency and high availability. Scaling is not just about adding more servers — it is about ensuring the system remains correct, observable, and secure as load increases.

Key challenges when scaling an API gateway include:

• Stateful concerns like rate limiting and authentication that require shared state across instances
• Connection management and keep-alive tuning to backend services
• Consistent configuration deployment across multiple instances
• Graceful shutdown and zero-downtime deployments
• Observability across a distributed fleet of gateway nodes

Phase 1: The Prototype Gateway

Let us start with a simple prototype. We will use Node.js with Express to build a minimal gateway that routes requests to two backend services: a user service and an order service.

// gateway.js - Prototype API Gateway
const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const app = express();
const PORT = process.env.PORT || 3000;

// Route user requests to the user service
app.use('/users', createProxyMiddleware({
  target: 'http://localhost:4001',
  changeOrigin: true,
  pathRewrite: { '^/users': '' }
}));

// Route order requests to the order service
app.use('/orders', createProxyMiddleware({
  target: 'http://localhost:4002',
  changeOrigin: true,
  pathRewrite: { '^/orders': '' }
}));

app.get('/health', (req, res) => {
  res.json({ status: 'ok' });
});

app.listen(PORT, () => {
  console.log(`API Gateway listening on port ${PORT}`);
});

This prototype works fine for development. It routes requests, rewrites paths, and exposes a health endpoint. However, it has no rate limiting, no caching, no authentication, and no horizontal scaling strategy. If this single process crashes, all traffic stops.

Phase 2: Adding Production Concerns

Authentication and Authorization

Production gateways typically validate JWT tokens before forwarding requests. Centralizing authentication at the gateway means backend services can trust incoming requests without reimplementing token validation.

// authMiddleware.js
const jwt = require('jsonwebtoken');

const SECRET = process.env.JWT_SECRET || 'dev-secret';

function authMiddleware(req, res, next) {
  const authHeader = req.headers.authorization;
  if (!authHeader || !authHeader.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'Missing or invalid token' });
  }

  const token = authHeader.split(' ')[1];
  try {
    const payload = jwt.verify(token, SECRET);
    req.user = payload;
    // Forward user identity to backend
    req.headers['x-user-id'] = payload.sub;
    req.headers['x-user-roles'] = payload.roles.join(',');
    next();
  } catch (err) {
    return res.status(401).json({ error: 'Invalid token' });
  }
}

module.exports = authMiddleware;

Apply this middleware selectively. Public endpoints like health checks should bypass authentication, while protected routes require it.

// gateway.js - with auth
const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');
const authMiddleware = require('./authMiddleware');

const app = express();

app.get('/health', (req, res) => res.json({ status: 'ok' }));

// Protected routes
app.use('/users', authMiddleware, createProxyMiddleware({
  target: 'http://localhost:4001',
  changeOrigin: true,
  pathRewrite: { '^/users': '' }
}));

app.use('/orders', authMiddleware, createProxyMiddleware({
  target: 'http://localhost:4002',
  changeOrigin: true,
  pathRewrite: { '^/orders': '' }
}));

app.listen(3000, () => console.log('Gateway on 3000'));

Rate Limiting with Redis

When you scale to multiple gateway instances, in-memory rate limiting is insufficient because each instance tracks its own counters independently. A client could bypass limits by hitting different instances. The solution is to use a shared data store like Redis.

// rateLimiter.js - Redis-backed token bucket
const redis = require('redis');

const client = redis.createClient({
  url: process.env.REDIS_URL || 'redis://localhost:6379'
});
client.connect();

const WINDOW_SECONDS = 60;
const MAX_REQUESTS = 100;

async function rateLimiter(req, res, next) {
  // Use API key or IP as the identifier
  const identifier = req.headers['x-api-key'] || req.ip;
  const key = `ratelimit:${identifier}`;
  const now = Date.now();

  try {
    const multi = client.multi();
    multi.zRemRangeByScore(key, 0, now - WINDOW_SECONDS * 1000);
    multi.zAdd(key, { score: now, value: `${now}` });
    multi.zCard(key);
    multi.expire(key, WINDOW_SECONDS);

    const results = await multi.exec();
    const requestCount = results[2];

    if (requestCount > MAX_REQUESTS) {
      res.set('Retry-After', WINDOW_SECONDS);
      return res.status(429).json({ error: 'Rate limit exceeded' });
    }

    res.set('X-RateLimit-Limit', MAX_REQUESTS);
    res.set('X-RateLimit-Remaining', Math.max(0, MAX_REQUESTS - requestCount));
    next();
  } catch (err) {
    console.error('Rate limiter error:', err);
    // Fail open in production to avoid blocking all traffic
    next();
  }
}

module.exports = rateLimiter;

This sliding window implementation uses Redis sorted sets to track request timestamps. It is accurate and works across any number of gateway instances since they all share the same Redis store.

Phase 3: Horizontal Scaling

Running Multiple Gateway Instances

To scale horizontally, run multiple instances of the gateway behind a load balancer. Each instance is stateless (state lives in Redis), so any instance can handle any request. Here is a Docker Compose setup that runs three gateway instances with an NGINX load balancer in front.

# docker-compose.yml
version: '3.8'

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  user-service:
    build: ./user-service
    ports:
      - "4001:4001"

  order-service:
    build: ./order-service
    ports:
      - "4002:4002"

  gateway-1:
    build: ./gateway
    environment:
      - PORT=3000
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=production-secret-change-me
    depends_on:
      - redis
      - user-service
      - order-service

  gateway-2:
    build: ./gateway
    environment:
      - PORT=3000
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=production-secret-change-me
    depends_on:
      - redis
      - user-service
      - order-service

  gateway-3:
    build: ./gateway
    environment:
      - PORT=3000
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=production-secret-change-me
    depends_on:
      - redis
      - user-service
      - order-service

  loadbalancer:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - gateway-1
      - gateway-2
      - gateway-3

NGINX Load Balancer Configuration

# nginx.conf
events {
    worker_connections 4096;
}

http {
    upstream gateway_backend {
        least_conn;
        server gateway-1:3000 max_fails=3 fail_timeout=30s;
        server gateway-2:3000 max_fails=3 fail_timeout=30s;
        server gateway-3:3000 max_fails=3 fail_timeout=30s;
        keepalive 64;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://gateway_backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Connection timeouts
            proxy_connect_timeout 5s;
            proxy_send_timeout 30s;
            proxy_read_timeout 30s;
        }

        location /health {
            access_log off;
            return 200 "ok\n";
        }
    }
}

The least_conn directive sends requests to the instance with the fewest active connections, which provides better distribution than round-robin when request processing times vary. The keepalive directive maintains persistent connections to the upstream gateway instances, reducing connection overhead.

Connection Pooling to Backends

A common scaling bottleneck is the number of connections between the gateway and backend services. Without connection pooling, every request opens a new TCP connection, which is expensive. Configure the proxy middleware to reuse connections.

// gateway.js - with connection pooling
const { createProxyMiddleware } = require('http-proxy-middleware');

const agent = require('http').globalAgent;
agent.maxSockets = 100; // Increase from default of 5

const userProxy = createProxyMiddleware({
  target: 'http://user-service:4001',
  changeOrigin: true,
  pathRewrite: { '^/users': '' },
  agent: agent,
  proxyTimeout: 10000,
  proxyTable: {
    // Optional: route to different backends based on request
    'multitenant.acme.com': 'http://user-service-tenant:4001'
  }
});

Phase 4: Caching for Performance

Caching is one of the most effective ways to scale an API gateway. Responses that are expensive to compute and change infrequently can be cached at the gateway level, reducing load on backend services dramatically.

// cache.js - Redis-backed response cache
const redis = require('redis');

const client = redis.createClient({
  url: process.env.REDIS_URL || 'redis://localhost:6379'
});
client.connect();

const DEFAULT_TTL = 60; // seconds

function createCache(options = {}) {
  const ttl = options.ttl || DEFAULT_TTL;
  const keyPrefix = options.keyPrefix || 'cache:';

  return async function cacheMiddleware(req, res, next) {
    // Only cache GET requests
    if (req.method !== 'GET') return next();

    const cacheKey = `${keyPrefix}${req.originalUrl}`;

    try {
      const cached = await client.get(cacheKey);
      if (cached) {
        const parsed = JSON.parse(cached);
        res.set('X-Cache', 'HIT');
        res.set('Cache-Control', `public, max-age=${ttl}`);
        return res.status(parsed.status).json(parsed.body);
      }

      // Intercept response to cache it
      const originalSend = res.json.bind(res);
      res.json = async function(body) {
        res.set('X-Cache', 'MISS');
        try {
          await client.setEx(cacheKey, ttl, JSON.stringify({
            status: res.statusCode,
            body: body
          }));
        } catch (err) {
          console.error('Cache write error:', err);
        }
        return originalSend(body);
      };

      next();
    } catch (err) {
      console.error('Cache read error:', err);
      next();
    }
  };
}

module.exports = createCache;

Use cache selectively. User-specific data should not be cached at the gateway level, or should be cached with a key that includes the user identifier. Public, read-heavy endpoints like product catalogs or configuration data are ideal candidates.

// Apply cache to specific routes
const createCache = require('./cache');

app.use('/products', createCache({ ttl: 300 }), productProxy);
app.use('/users', authMiddleware, userProxy); // No cache for user data

Phase 5: Observability

Structured Logging

In a multi-instance setup, logs from all gateway instances must be aggregated and searchable. Use structured JSON logging with correlation IDs so you can trace a request across the load balancer, gateway, and backend services.

// logger.js
const winston = require('winston');
const { v4: uuidv4 } = require('uuid');

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  defaultMeta: {
    service: 'api-gateway',
    version: process.env.APP_VERSION || '1.0.0'
  },
  transports: [
    new winston.transports.Console()
  ]
});

function requestLogger(req, res, next) {
  const requestId = req.headers['x-request-id'] || uuidv4();
  req.requestId = requestId;
  res.set('X-Request-Id', requestId);

  const startTime = process.hrtime.bigint();

  res.on('finish', () => {
    const durationMs = Number(process.hrtime.bigint() - startTime) / 1e6;
    logger.info('request completed', {
      requestId,
      method: req.method,
      path: req.path,
      status: res.statusCode,
      durationMs: Math.round(durationMs * 100) / 100,
      userAgent: req.headers['user-agent'],
      ip: req.ip
    });
  });

  next();
}

module.exports = { logger, requestLogger };

Metrics with Prometheus

Logs tell you what happened; metrics tell you what is happening in aggregate. Expose Prometheus metrics so you can build dashboards and alerts.

// metrics.js
const promClient = require('prom-client');

const collectDefaultMetrics = promClient.collectDefaultMetrics;
collectDefaultMetrics({ register: promClient.register });

const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
});

const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status']
});

function metricsMiddleware(req, res, next) {
  const startTime = process.hrtime();

  res.on('finish', () => {
    const [seconds, nanoseconds] = process.hrtime(startTime);
    const duration = seconds + nanoseconds / 1e9;
    const route = req.route ? req.route.path : req.path;

    httpRequestDuration
      .labels(req.method, route, res.statusCode.toString())
      .observe(duration);

    httpRequestsTotal
      .labels(req.method, route, res.statusCode.toString())
      .inc();
  });

  next();
}

module.exports = { metricsMiddleware, promClient };

Expose the metrics endpoint on a separate port or path so it is not accessible to external clients.

// gateway.js - metrics endpoint
const { metricsMiddleware, promClient } = require('./metrics');

app.use(metricsMiddleware);

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

Phase 6: Graceful Shutdown and Zero-Downtime Deploys

When scaling, you will frequently deploy new versions of the gateway. Without graceful shutdown, in-flight requests are dropped during deploys. The gateway must stop accepting new connections, finish processing existing requests, and then exit.

// gracefulShutdown.js
function setupGracefulShutdown(server, options = {}) {
  const timeout = options.timeout || 30000;

  function shutdown(signal) {
    console.log(`Received ${signal}, shutting down gracefully`);

    // Stop accepting new connections
    server.close(() => {
      console.log('All connections closed, exiting');
      process.exit(0);
    });

    // Force exit after timeout
    setTimeout(() => {
      console.error('Forcing shutdown after timeout');
      process.exit(1);
    }, timeout).unref();
  }

  process.on('SIGTERM', () => shutdown('SIGTERM'));
  process.on('SIGINT', () => shutdown('SIGINT'));
}

module.exports = setupGracefulShutdown;

// gateway.js - with graceful shutdown
const server = app.listen(PORT, () => {
  console.log(`API Gateway listening on port ${PORT}`);
});

const setupGracefulShutdown = require('./gracefulShutdown');
setupGracefulShutdown(server, { timeout: 30000 });

Combine graceful shutdown with the load balancer health checks. When a gateway instance starts shutting down, it should fail the load balancer health check so traffic stops being routed to it. The NGINX max_fails and fail_timeout settings we configured earlier handle this automatically.

Phase 7: Autoscaling

With multiple stateless instances behind a load balancer, you can autoscale based on metrics. If you are running on Kubernetes, use the Horizontal Pod Autoscaler to scale based on CPU or custom metrics.

# gateway-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-gateway-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-gateway
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "500"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60

The stabilizationWindowSeconds for scale-down prevents flapping — the system waits 5 minutes of low load before removing instances. The scale-up policy allows doubling capacity every 30 seconds to handle traffic spikes quickly.

Best Practices

• Keep the gateway stateless. All shared state (rate limits, cache, sessions) should live in Redis or another external store. This makes horizontal scaling trivial.
• Fail open for non-critical middleware. If Redis goes down, your rate limiter should allow traffic through rather than blocking all requests. Blocking all traffic is usually worse than allowing a few extra requests.
• Use circuit breakers for backend calls. If a backend service is failing, stop sending traffic to it temporarily rather than letting requests pile up and exhaust connections.
• Set aggressive timeouts. Every external call should have a timeout. A slow backend should not hold gateway resources indefinitely. Set connect, read, and write timeouts at every layer.
• Version your API at the gateway. Route /v1/users and /v2/users to different backends. This lets you roll out new API versions without disrupting existing clients.
• Compress responses. Enable gzip or brotli compression at the gateway. This reduces bandwidth and improves client-perceived latency, especially for JSON-heavy APIs.
• Use connection keep-alive everywhere. Between client and load balancer, load balancer and gateway, and gateway and backends. Connection establishment is expensive at scale.
• Monitor the four golden signals. Track latency, traffic, errors, and saturation. Set alerts on error rate and p99 latency, not just averages.
• Test with production-like load. Use tools like k6, Locust, or Artillery to load test before going to production. A gateway that handles 100 RPS in dev may behave very differently at 10,000 RPS.
• Secure your configuration. Never hardcode secrets. Use environment variables or a secrets manager. Rotate JWT signing keys regularly and support multiple valid keys during rotation.

Circuit Breaker Example

// circuitBreaker.js
const circuitBreaker = require('opossum');

function createCircuitBreaker(proxyFn, options = {}) {
  return new circuitBreaker(proxyFn, {
    timeout: options.timeout || 5000,
    errorThresholdPercentage: options.errorThreshold || 50,
    resetTimeout: options.resetTimeout || 30000,
    rollingCountTimeout: 60000,
    rollingCountBuckets: 10
  });
}

// Usage in a route handler
const breaker = createCircuitBreaker(async (req, res) => {
  // Forward to backend
  return forwardRequest(req, res);
});

app.use('/orders', async (req, res) => {
  try {
    await breaker.fire(req, res);
  } catch (err) {
    if (breaker.opened) {
      return res.status(503).json({
        error: 'Service temporarily unavailable',
        retryAfter: 30
      });
    }
    res.status(500).json({ error: 'Internal error' });
  }
});

// Listen to circuit events for observability
breaker.on('open', () => logger.warn('Circuit breaker opened for orders'));
breaker.on('close', () => logger.info('Circuit breaker closed for orders'));
breaker.on('fallback', () => logger.warn('Circuit breaker fallback triggered'));

Conclusion

Scaling an API gateway from prototype to production is a journey through multiple layers of concern. The prototype proves the routing concept; production hardens it with authentication, rate limiting, caching, observability, graceful shutdown, and autoscaling. The key principles are to keep the gateway stateless, push shared state to Redis, use a load balancer for horizontal scaling, and invest heavily in observability so you can understand system behavior under load. By following the patterns and code examples in this tutorial, you can build an API gateway that handles production traffic reliably, scales elastically with demand, and remains maintainable as your system grows. Remember that scaling is iterative — start with the basics, measure, and add complexity only when your metrics tell you it is necessary.