← Back to DevBytes
Developer Tutorial August 19, 2026 11 min read

Auto-scaling LLM Inference with KEDA and Prometheus

Auto-scaling LLM Inference with KEDA and Prometheus

Large Language Model (LLM) inference workloads are notoriously bursty and resource-intensive. Unlike traditional web services, LLM inference endpoints can experience sudden spikes in request volume, long-tail latencies, and GPU memory pressure that make static scaling strategies inefficient. This tutorial walks you through building a robust auto-scaling system for LLM inference using KEDA (Kubernetes Event-Driven Autoscaling) and Prometheus, enabling your inference servers to scale based on real workload signals rather than crude CPU or memory thresholds.

What Is KEDA?

KEDA is a Kubernetes-based event-driven autoscaler that extends the standard Horizontal Pod Autoscaler (HPA). While the native HPA can only scale based on CPU and memory metrics, KEDA introduces a concept called Scalers — pluggable metric sources that include message queues, databases, and monitoring systems like Prometheus. KEDA acts as a bridge: it collects external metrics, feeds them to the Kubernetes metrics API, and lets the HPA consume them for scaling decisions.

The key advantage for LLM workloads is that KEDA can scale based on custom business metrics such as active inference requests, queue depth, tokens-per-second throughput, or average request latency — metrics that actually reflect the health and load of an inference server.

Why Auto-scaling LLM Inference Matters

LLM inference is expensive. GPUs are scarce, costly, and often underutilized when traffic is low. At the same time, under-provisioning during traffic spikes leads to request timeouts, degraded user experience, and potential SLA violations. Consider these challenges:

By combining KEDA with Prometheus, you can scale based on metrics that directly represent inference load — such as the number of concurrent requests in the inference queue — giving you precise, responsive autoscaling that minimizes cost while maintaining performance.

Architecture Overview

The system consists of four main components working together:

The data flow is: Inference Server → Prometheus → KEDA → HPA → Kubernetes Scheduler → new pods.

Prerequisites

Before you begin, ensure you have the following:

Step 1: Install Prometheus

If you don't already have Prometheus running, install it using the Prometheus community Helm chart. We'll use the kube-prometheus-stack which includes Prometheus, Grafana, and alerting components.

# Add the Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack in the monitoring namespace
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

Verify that Prometheus is running:

kubectl get pods -n monitoring

You should see Prometheus, Grafana, and alertmanager pods in a running state.

Step 2: Install KEDA

Install KEDA using its official Helm chart. KEDA will register itself as a custom metrics API server in your cluster.

# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
helm repo update

# Install KEDA in the keda namespace
helm install keda kedacore/keda \
  --namespace keda \
  --create-namespace

Verify the KEDA installation:

kubectl get pods -n keda

You should see the KEDA operator and metrics API server pods running.

Step 3: Deploy an LLM Inference Server with Metrics

For this tutorial, we'll deploy a vLLM inference server. vLLM exposes useful Prometheus metrics out of the box, including vllm:num_requests_running, vllm:num_requests_waiting, and vllm:gpu_cache_usage_perc. These metrics are perfect for autoscaling decisions.

First, create a Kubernetes deployment for the inference server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: llm-serving
  labels:
    app: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - --model
        - meta-llama/Llama-2-7b-chat-hf
        - --port
        - "8000"
        - --tensor-parallel-size
        - "1"
        - --max-model-len
        - "4096"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 16Gi
          requests:
            nvidia.com/gpu: 1
            memory: 8Gi
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 30

Create a Service to expose the inference server and a ServiceMonitor so Prometheus can scrape its metrics:

apiVersion: v1
kind: Service
metadata:
  name: llm-inference
  namespace: llm-serving
spec:
  selector:
    app: llm-inference
  ports:
  - port: 8000
    targetPort: 8000
    name: http
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-inference-monitor
  namespace: llm-serving
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: llm-inference
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

Apply these manifests:

kubectl create namespace llm-serving
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Step 4: Verify Metrics in Prometheus

Before configuring KEDA, verify that Prometheus is successfully scraping the inference server metrics. Port-forward to the Prometheus UI:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090

Open your browser to http://localhost:9090 and query for the waiting requests metric:

vllm:num_requests_waiting

If you see metric values returned, Prometheus is successfully scraping your inference server. If not, check that the ServiceMonitor has the correct label selector and that the inference server pod is healthy.

Step 5: Create a KEDA ScaledObject

Now we configure KEDA to scale the inference deployment based on the number of waiting requests in the vLLM queue. The ScaledObject is KEDA's custom resource that defines the scaling behavior.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    name: llm-inference
    kind: Deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 15
  cooldownPeriod: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      threshold: "5"
      query: |
        sum(vllm:num_requests_waiting{namespace="llm-serving"})
      activationThreshold: "1"

Let's break down the key fields in this ScaledObject:

Apply the ScaledObject:

kubectl apply -f scaledobject.yaml

Verify that KEDA has created the underlying HPA:

kubectl get hpa -n llm-serving

You should see an HPA named keda-hpa-llm-inference with the target metric showing the current Prometheus value.

Step 6: Multi-Metric Scaling Strategy

Scaling on a single metric is a good start, but production LLM serving benefits from a multi-dimensional approach. Let's create a more sophisticated ScaledObject that considers both queue depth and GPU cache utilization:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    name: llm-inference
    kind: Deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 15
  cooldownPeriod: 60
  triggers:
  - type: prometheus
    name: waiting-requests
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      threshold: "5"
      query: |
        sum(vllm:num_requests_waiting{namespace="llm-serving"})
  - type: prometheus
    name: running-requests
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_running
      threshold: "20"
      query: |
        sum(vllm:num_requests_running{namespace="llm-serving"})
  - type: prometheus
    name: p99-latency
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
      metricName: vllm_request_latency_p99
      threshold: "5000"
      query: |
        histogram_quantile(0.99,
          sum(rate(vllm:request_latency_seconds_bucket{namespace="llm-serving"}[1m]))
          by (le)
        ) * 1000

When multiple triggers are defined, KEDA computes the desired replica count for each trigger independently and uses the maximum value. This means if any single metric indicates that more replicas are needed, KEDA will scale up. This is the safest approach for latency-sensitive workloads.

Step 7: Testing the Autoscaling Behavior

To verify that autoscaling works, generate load against your inference server and watch the replicas scale. First, set up a port-forward to the inference service:

kubectl port-forward -n llm-serving svc/llm-inference 8000:8000

In a separate terminal, use a load testing tool to send concurrent requests. Here's a Python script using asyncio and aiohttp to simulate concurrent chat completions:

import asyncio
import aiohttp
import json
import time

API_URL = "http://localhost:8000/v1/chat/completions"
CONCURRENT_REQUESTS = 50
TOTAL_REQUESTS = 500

async def send_request(session, request_id):
    payload = {
        "model": "meta-llama/Llama-2-7b-chat-hf",
        "messages": [
            {"role": "user", "content": "Explain quantum computing in 200 words."}
        ],
        "max_tokens": 256,
        "temperature": 0.7
    }
    try:
        async with session.post(API_URL, json=payload) as response:
            result = await response.json()
            print(f"Request {request_id}: status={response.status}")
            return response.status
    except Exception as e:
        print(f"Request {request_id}: error={e}")
        return None

async def main():
    connector = aiohttp.TCPConnector(limit=CONCURRENT_REQUESTS)
    async with aiohttp.ClientSession(connector=connector) as session:
        semaphore = asyncio.Semaphore(CONCURRENT_REQUESTS)

        async def bounded_request(req_id):
            async with semaphore:
                return await send_request(session, req_id)

        tasks = [bounded_request(i) for i in range(TOTAL_REQUESTS)]
        start = time.time()
        results = await asyncio.gather(*tasks)
        elapsed = time.time() - start

        success = sum(1 for r in results if r == 200)
        print(f"\nCompleted {success}/{TOTAL_REQUESTS} requests in {elapsed:.1f}s")

asyncio.run(main())

While the load test runs, monitor the scaling behavior in another terminal:

# Watch the deployment scale
kubectl get deployment llm-inference -n llm-serving -w

# Watch the HPA status
kubectl get hpa -n llm-serving -w

# Check the current queue depth in Prometheus
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=vllm:num_requests_waiting'

You should observe the replica count increasing as the queue depth rises, then decreasing after the load test completes and the cooldown period elapses.

Best Practices

Choose the Right Scaling Metric

The most effective scaling metric for LLM inference is typically the number of waiting requests in the inference queue. This directly represents unmet demand. CPU and memory metrics are poor proxies because GPU-bound inference workloads often show low CPU utilization even when the system is saturated. Avoid scaling on throughput metrics alone, as they don't capture latency degradation under load.

Set Appropriate Thresholds

Thresholds should reflect your latency SLOs. If your SLO is p99 latency under 5 seconds, set the waiting-request threshold low enough that requests don't queue for long. A threshold of 3-5 waiting requests per replica is a reasonable starting point. Monitor the relationship between queue depth and latency, then tune accordingly.

Account for Cold Starts

Loading model weights into GPU memory takes time. To mitigate cold start latency, consider these strategies:

Prevent Scaling Flapping

Rapid scaling up and down wastes resources and causes instability. Use a cooldownPeriod of 60-120 seconds to prevent premature scale-downs. You can also use Prometheus recording rules to smooth out noisy metrics:

# Add to Prometheus rules
groups:
- name: llm_inference
  rules:
  - record: llm:waiting_requests_avg_1m
    expr: avg_over_time(vllm:num_requests_waiting[1m])
  - record: llm:running_requests_avg_1m
    expr: avg_over_time(vllm:num_requests_running[1m])

Then reference the smoothed metric in your KEDA trigger query for more stable scaling decisions.

Handle GPU Resource Constraints

Each LLM inference replica requires a GPU, so your maxReplicaCount should never exceed your available GPU capacity. Use node labels and taints to ensure inference pods are scheduled on GPU nodes. Consider using cluster autoscaler or Karpenter in combination with KEDA to provision additional GPU nodes when needed:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    name: llm-inference
    kind: Deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 15
  cooldownPeriod: 120
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 50
            periodSeconds: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
      metricName: llm_waiting_requests_avg
      threshold: "5"
      query: |
        llm:waiting_requests_avg_1m{namespace="llm-serving"}

The advanced.horizontalPodAutoscalerConfig section lets you fine-tune the HPA behavior. The scale-up configuration allows doubling replicas every 30 seconds for fast response to spikes, while the scale-down configuration removes at most 50% of replicas per minute with a 5-minute stabilization window.

Monitor the Autoscaler Itself

KEDA exposes its own Prometheus metrics that you should monitor. Key metrics include:

Create Grafana dashboards that visualize the relationship between queue depth, replica count, and request latency. This visibility is essential for tuning thresholds and understanding scaling behavior over time.

Use Scale-to-Zero Carefully

KEDA supports scaling to zero replicas when there's no traffic, which can save significant GPU costs. However, for LLM inference, the cold start penalty is severe. If you do enable scale-to-zero, combine it with a scheduled trigger that pre-warms replicas before expected traffic peaks:

triggers:
- type: prometheus
  metadata:
    serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
    metricName: vllm_num_requests_waiting
    threshold: "5"
    query: |
      sum(vllm:num_requests_waiting{namespace="llm-serving"})
- type: cron
  metadata:
    timezone: America/New_York
    start: "0 8 * * 1-5"
    end: "0 18 * * 1-5"
    desiredReplicas: "2"

This configuration scales to zero during off-hours but ensures at least 2 warm replicas are running during business hours on weekdays.

Conclusion

Auto-scaling LLM inference with KEDA and Prometheus gives you fine-grained, metric-driven control over your GPU infrastructure. By scaling on meaningful signals like queue depth and request latency rather than generic CPU metrics, you can maintain low latency during traffic spikes while minimizing costs during quiet periods. The combination of KEDA's flexible trigger system and Prometheus's powerful query language makes this approach adaptable to virtually any inference framework — whether you're running vLLM, Text Generation Inference, or a custom serving solution. Start with a single metric and conservative thresholds, then iteratively refine your scaling configuration based on observed behavior. With proper tuning, this setup will keep your LLM inference endpoints responsive and cost-efficient under any load pattern.

— Ad —

Google AdSense will appear here after approval

← Back to all articles