Fix Kubernetes 'CrashLoopBackOff' Error in Production: Root Cause Analysis

Understanding CrashLoopBackOff in Kubernetes

When a container inside a Kubernetes pod repeatedly starts, crashes, and then restarts in a vicious cycle, Kubernetes enters a back-off state known as CrashLoopBackOff. This is not the root cause itself — it is a protective mechanism. Kubernetes detects that your container keeps exiting with a non-zero exit code (or being killed by the OOM killer, a liveness probe failure, or a start probe failure) and progressively increases the delay between restart attempts to prevent overwhelming the node. The delay follows an exponential back-off pattern: 10 seconds, 20 seconds, 40 seconds, 80 seconds, and so on, up to a maximum of 5 minutes between restarts.

For developers and platform engineers, seeing CrashLoopBackOff in the output of kubectl get pods triggers a familiar sense of urgency. The pod status tells you that something is fundamentally broken, but it gives you zero information about what is broken. Effective root cause analysis (RCA) requires a systematic approach that peels back the layers — from the pod status, to container exit codes, to application logs, to resource constraints, to configuration errors — until you isolate the exact failure.

Why CrashLoopBackOff Matters in Production

In a production environment, a CrashLoopBackOff state is more dangerous than a simple failed pod. Here's why:

• Cascading failures: If the crashing pod is part of a Deployment or StatefulSet that serves live traffic, each restart cycle causes a brief window where the container is not ready. This can lead to increased latency, dropped connections, and 5xx errors for downstream consumers.
• Resource exhaustion: Repeatedly starting containers that allocate memory, open file descriptors, or establish connections before crashing can slowly drain node resources, affecting co-located pods.
• Masked root causes: The back-off delay can obscure the actual failure pattern. An issue that happens every 5 minutes is harder to catch than one that happens every second.
• Data corruption risk: For stateful workloads like databases, repeated unclean shutdowns can corrupt write-ahead logs, trigger lengthy recovery processes on the next start, and in extreme cases cause permanent data loss.
• Alert fatigue: Without proper RCA, teams may respond to the symptom (restarting the pod or scaling up) rather than fixing the underlying bug, leading to recurring incidents.

The Root Cause Analysis Framework

Before diving into specific fixes, establish a structured RCA process. This prevents the common trap of guessing, applying random fixes, and hoping the problem disappears. Follow these steps in order:

1. Observe the pod status and restart count.
2. Inspect the container exit code and termination reason.
3. Retrieve container logs (both current and previous crashed instances).
4. Examine Kubernetes events for the pod.
5. Validate resource limits, probes, and security contexts.
6. Reproduce the crash in a controlled environment if needed.

Each step builds on the previous one. Jumping straight to logs without checking the exit code first can waste time if the container was killed by the OOM killer — in that case, the application never had a chance to log anything.

Step 1: Gather Baseline Information

Start with the broadest view possible. List all pods in the affected namespace and note any that show CrashLoopBackOff or a high restart count:

kubectl get pods -n production -o wide

Expected output for a troubled pod might look like:

NAME                          READY   STATUS             RESTARTS   AGE
api-gateway-7d4f8b9c6-5xm3k   0/1     CrashLoopBackOff   42         2h
payment-svc-5c8b9f7d4-k9pqw   1/1     Running            0          3d
redis-master-0                1/1     Running            0          7d

Pay attention to the RESTARTS column. A high number over a short period indicates a rapid crash loop. Next, get detailed information about the specific pod:

kubectl describe pod api-gateway-7d4f8b9c6-5xm3k -n production

Scroll through the output and locate the Containers section, specifically the Last State block. This is your single most valuable clue:

Containers:
  api-gateway:
    Container ID:   containerd://a1b2c3d4e5f6...
    Image:          registry.example.com/api-gateway:v2.4.1
    Image ID:       registry.example.com/api-gateway@sha256:abc123...
    Port:           8080/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      2025-03-15T08:22:10Z
      Finished:     2025-03-15T08:22:13Z
    Ready:          False
    Restart Count:  42

Step 2: Decode the Exit Code

The exit code is the first concrete clue. Here is a reference for common exit codes and what they imply:

• Exit Code 0: The container completed successfully and exited. If you see this in a CrashLoopBackOff, your container's main process is exiting normally when it shouldn't — perhaps a script that reaches the end, or a command that runs once and stops.
• Exit Code 1: Application error. The process inside the container encountered an unhandled exception, a fatal error, or called exit(1). This is the most common exit code and requires log analysis.
• Exit Code 2: Typically a shell syntax error or a command misuse (e.g., trying to execute a directory). Check your entrypoint script.
• Exit Code 126: Command invoked is not executable (permission problem). Check file permissions inside the container.
• Exit Code 127: Command not found. The binary or script specified in the entrypoint or CMD does not exist in the container's PATH.
• Exit Code 137: The container was killed by a signal — specifically SIGKILL (signal 9). On Kubernetes, this almost always means the container exceeded its memory limit and was killed by the OOM killer. Add 128 to the signal number to get the exit code (128 + 9 = 137).
• Exit Code 139: Segmentation fault (signal 11). The application tried to access invalid memory. This indicates a bug in the application code (128 + 11 = 139).
• Exit Code 143: The container received SIGTERM (signal 15) and exited gracefully. This is normal during pod deletion. If you see this in CrashLoopBackOff, the pod may be receiving external termination signals repeatedly (128 + 15 = 143).

If the Reason field shows OOMKilled instead of Error, you have a clear memory problem:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      2025-03-15T08:22:10Z
  Finished:     2025-03-15T08:22:13Z

Step 3: Extract Container Logs

For exit code 1 (application error), logs are essential. Use kubectl logs to get the current container's stdout/stderr. However, since the container is in a crash loop, the current instance may not have run long enough to produce useful logs. Instead, retrieve logs from the previous crashed instance:

# Get logs from the currently running (or waiting) container
kubectl logs api-gateway-7d4f8b9c6-5xm3k -n production

# Get logs from the PREVIOUS crashed container instance — this is usually what you need
kubectl logs api-gateway-7d4f8b9c6-5xm3k -n production --previous

# If the pod has multiple containers, specify the container name
kubectl logs api-gateway-7d4f8b9c6-5xm3k -n production -c api-gateway --previous

# Stream logs in real-time (useful if the crash takes a while to occur)
kubectl logs -f api-gateway-7d4f8b9c6-5xm3k -n production

For pods with init containers, check their logs too — a failing init container will prevent the main container from ever starting:

kubectl logs api-gateway-7d4f8b9c6-5xm3k -n production -c init-db-migration --previous

Step 4: Examine Kubernetes Events

Kubernetes events provide a timeline of what happened to the pod. They capture liveness probe failures, OOM kills, scheduling failures, and image pull errors:

# Get events for the specific pod
kubectl describe pod api-gateway-7d4f8b9c6-5xm3k -n production | grep -A 30 "Events:"

# Or get all recent events in the namespace, sorted by time
kubectl get events -n production --sort-by='.lastTimestamp' | tail -20

# Filter events related to a specific pod
kubectl get events -n production --field-selector involvedObject.name=api-gateway-7d4f8b9c6-5xm3k

Look for patterns like:

Events:
  Type     Reason     Age                    From                Message
  ----     ------     ----                   ----                -------
  Normal   Scheduled  10m                    default-scheduler   Successfully assigned production/api-gateway-7d4f8b9c6-5xm3k to node-worker-3
  Normal   Pulled     9m30s                  kubelet             Container image "registry.example.com/api-gateway:v2.4.1" already present on machine
  Warning  BackOff    9m                     kubelet             Back-off restarting failed container
  Normal   Created    8m50s (x5 over 9m30s)  kubelet             Created container api-gateway
  Normal   Started    8m50s (x5 over 9m30s)  kubelet             Started container api-gateway
  Warning  Failed     8m49s (x5 over 9m29s)  kubelet             Error: container exited with exit code 1

Common Root Causes and Their Fixes

Cause 1: Application Bug (Exit Code 1)

The application itself is crashing due to an unhandled exception, a missing dependency, or a fatal configuration error. The stack trace in the logs will pinpoint the exact line of code.

Example log output:

Error: Cannot find module '/app/node_modules/critical-package'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1021:15)
    at Function.Module._load (node:internal/modules/cjs/loader:866:27)
Fatal error: exit code 1

Fix: Roll back to a known-good image version immediately, then debug the application in a staging environment:

# Roll back the Deployment to the previous revision
kubectl rollout undo deployment/api-gateway -n production

# Check rollout history to identify the problematic revision
kubectl rollout history deployment/api-gateway -n production

# Roll back to a specific revision
kubectl rollout undo deployment/api-gateway -n production --to-revision=3

Then, reproduce the crash locally using the exact same container image:

# Pull the problematic image and run it locally with the same environment variables
docker run --rm -it \
  -e DATABASE_URL="postgres://user:pass@host:5432/db" \
  -e LOG_LEVEL="debug" \
  registry.example.com/api-gateway:v2.4.1

Cause 2: OOMKilled (Exit Code 137)

The container exceeded its memory limit. Kubernetes sends SIGKILL, which cannot be caught by the application. The process dies instantly with no chance to log a stack trace.

Diagnostic indicators:

• Exit code is 137 and Reason is OOMKilled in kubectl describe.
• Logs end abruptly with no error message — they simply stop.
• Node-level dmesg shows OOM killer activity.

Check node-level OOM events:

# SSH into the node or use a privileged daemonset to check kernel logs
dmesg | grep -i "out of memory" | tail -10
dmesg | grep -i "oom" | tail -10

# Or use kubectl node-shell if you have the plugin installed
kubectl node-shell node-worker-3
dmesg | grep -i oom

Fix options:

1. Increase the memory limit if the application legitimately needs more memory:

# Edit the Deployment resource limits
kubectl edit deployment api-gateway -n production

# In the editor, find the resources section and increase limits.memory
# Example: from 256Mi to 512Mi or 1Gi
spec:
  containers:
  - name: api-gateway
    resources:
      limits:
        memory: "512Mi"   # was "256Mi"
      requests:
        memory: "256Mi"   # was "128Mi"

2. Fix a memory leak in the application code. Look for unbounded caches, unclosed connections, or goroutine leaks.
3. Add a memory request that matches typical usage so the scheduler places the pod on a node with sufficient capacity:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

4. Use a liveness probe with a grace period to allow the application to handle increased memory pressure gracefully before being killed. Note: this does not prevent OOM kills, but it can help with application-level memory management:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3
terminationGracePeriodSeconds: 30

Cause 3: Liveness Probe Failure

The application is running, but the liveness probe consistently fails. Kubernetes restarts the container because it considers it unhealthy, even though the process is technically alive.

Diagnostic indicators:

• In kubectl describe, you see Liveness probe failed events.
• The container's Last State shows Reason: Error with exit code 143 (SIGTERM from the kubelet stopping the container).
• Logs show the application is running normally, but the health check endpoint returns a non-200 status.

Events:
  Warning  Unhealthy  45s (x3 over 75s)  kubelet  Liveness probe failed: Get "http://10.244.2.15:8080/healthz": dial tcp 10.244.2.15:8080: connect: connection refused

Common liveness probe mistakes:

• Probe checking an external dependency: If your /healthz endpoint checks database connectivity and the database is temporarily slow, the probe fails and Kubernetes restarts your perfectly healthy application.
• Probe timeout too short: The application needs 5 seconds to respond under load, but the probe timeout is set to 1 second.
• initialDelaySeconds too short: The application needs 60 seconds to start, but the probe starts checking after 10 seconds.

Fix — redesign the health check endpoints:

livenessProbe:
  httpGet:
    path: /healthz           # This should ONLY check the process itself — memory, event loop, etc.
    port: 8080
  initialDelaySeconds: 60   # Wait long enough for the app to fully initialize
  periodSeconds: 10
  timeoutSeconds: 5          # Reasonable timeout
  failureThreshold: 5        # Allow more failures before restarting — prevents flapping

readinessProbe:
  httpGet:
    path: /ready             # This CAN check external dependencies (DB, cache, etc.)
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3

Inside the application, keep the liveness endpoint cheap and dependency-free:

// Good: liveness checks only internal state
app.get('/healthz', (req, res) => {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
});

// Bad: liveness checks database — causes cascade restarts
app.get('/healthz', async (req, res) => {
  try {
    await db.query('SELECT 1');
    res.status(200).json({ status: 'ok' });
  } catch (e) {
    res.status(500).json({ status: 'error' });  // This kills the pod!
  }
});

Cause 4: Missing Dependencies or Configuration (Exit Code 2, 126, 127)

The container's entrypoint references a binary, script, or file that does not exist or is not executable. This is common when the Dockerfile has a multi-stage build and the final image is missing expected artifacts.

Example log output:

/bin/sh: /app/start.sh: not found
# or
exec: "/app/server": permission denied
# or
Error: /app/config.yaml: No such file or directory

Fix — validate the container image:

# Inspect the entrypoint and CMD of the image
docker inspect registry.example.com/api-gateway:v2.4.1 | jq '.[0].Config.Entrypoint'
docker inspect registry.example.com/api-gateway:v2.4.1 | jq '.[0].Config.Cmd'

# Run a shell inside the image to verify file existence and permissions
docker run --rm -it --entrypoint /bin/sh registry.example.com/api-gateway:v2.4.1

# Inside the container, check:
ls -la /app/
file /app/server
/app/server --version    # Test execution

If using ConfigMaps or Secrets mounted as files, verify they exist and are mounted correctly:

kubectl get configmap app-config -n production -o yaml
kubectl describe pod api-gateway-7d4f8b9c6-5xm3k -n production | grep -A 10 "Mounts:"

Cause 5: Start Probe Failure (Kubernetes 1.18+)

Start probes are designed for slow-starting applications. If a start probe fails, the container is killed before it finishes initializing. The symptoms look similar to liveness probe failures, but the probe type is different.

Events:
  Warning  Unhealthy  30s (x3 over 60s)  kubelet  Start probe failed: Get "http://10.244.2.15:8080/startup": context deadline exceeded

Fix — configure the start probe appropriately:

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 30   # Allow up to 300 seconds (5 minutes) for startup
  timeoutSeconds: 10

# Once the startup probe succeeds, the liveness probe takes over
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

Cause 6: Image Pull Errors

Sometimes the pod never reaches CrashLoopBackOff because it cannot even pull the image. But in some cases, transient image pull failures can cause a crash loop. Look for ImagePullBackOff or ErrImagePull in the pod status:

kubectl describe pod api-gateway-7d4f8b9c6-5xm3k -n production | grep -i "image"

# Common errors:
# - "Failed to pull image": registry authentication or network issue
# - "ImagePullBackOff": image doesn't exist or tag is wrong
# - "manifest unknown": the image tag was deleted from the registry

Fix:

# Verify the image exists and is accessible
skopeo inspect docker://registry.example.com/api-gateway:v2.4.1

# Check if imagePullSecrets are configured correctly
kubectl get pod api-gateway-7d4f8b9c6-5xm3k -n production -o jsonpath='{.spec.imagePullSecrets}'

# Create or update the registry secret if needed
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy-bot \
  --docker-password=your-token \
  --docker-email=deploy@example.com \
  -n production

# Reference it in the Deployment
kubectl patch deployment api-gateway -n production -p '
{
  "spec": {
    "template": {
      "spec": {
        "imagePullSecrets": [
          { "name": "regcred" }
        ]
      }
    }
  }
}'

Cause 7: Race Conditions with Init Containers

Init containers run to completion before the main container starts. If an init container fails, the main container never runs, and the pod enters Init:CrashLoopBackOff or Init:Error.

kubectl get pods -n production
NAME                          READY   STATUS                  RESTARTS   AGE
api-gateway-7d4f8b9c6-5xm3k   0/1     Init:CrashLoopBackOff   12         30m

Diagnose the init container:

kubectl logs api-gateway-7d4f8b9c6-5xm3k -n production -c init-db-migration --previous
kubectl describe pod api-gateway-7d4f8b9c6-5xm3k -n production | grep -A 20 "Init Containers:"

Example init container failure — database not ready:

# Init container log
Error: unable to connect to database: dial tcp 10.244.1.10:5432: connection refused
exit code 1

Fix — add retry logic to the init container script:

# In your init container script, retry with exponential backoff
#!/bin/sh
set -e

MAX_RETRIES=30
RETRY_DELAY=2

for i in $(seq 1 $MAX_RETRIES); do
  if psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -c "SELECT 1" > /dev/null 2>&1; then
    echo "Database is ready"
    exit 0
  fi
  echo "Waiting for database... attempt $i/$MAX_RETRIES"
  sleep $RETRY_DELAY
  RETRY_DELAY=$((RETRY_DELAY * 2 > 60 ? 60 : RETRY_DELAY * 2))
done

echo "Database did not become ready in time"
exit 1

Advanced Diagnostic Techniques

Using Ephemeral Debug Containers

For pods that crash too quickly to exec into, use an ephemeral container (Kubernetes 1.23+) to inspect the filesystem:

# Attach a debug container to a running (or crashing) pod
kubectl debug api-gateway-7d4f8b9c6-5xm3k -n production \
  --image=busybox:latest \
  --target=api-gateway \
  --container=debugger \
  -- sleep 3600

# Now exec into the debug container and inspect the target container's filesystem
kubectl exec -it api-gateway-7d4f8b9c6-5xm3k -n production -c debugger -- sh

# Inside the debug container, the target container's filesystem is at /proc/1/root
ls -la /proc/1/root/app/
cat /proc/1/root/app/config/config.yaml

Capturing Core Dumps for Exit Code 139 (Segfault)

If you see exit code 139, the application is segfaulting. Capture a core dump for offline analysis:

# Modify the Deployment to set core dump pattern and disable core size limits
# This requires a privileged container or appropriate security context
spec:
  containers:
  - name: api-gateway
    securityContext:
      privileged: true   # Needed to modify /proc/sys/kernel/core_pattern
    env:
    - name: GOTRACEBACK
      value: "crash"     # For Go applications
    - name: LLVM_PROFILE_FILE
      value: "/tmp/crash-%p.profraw"
    volumeMounts:
    - name: core-dumps
      mountPath: /tmp/coredumps
  volumes:
  - name: core-dumps
    hostPath:
      path: /var/coredumps
      type: DirectoryOrCreate

Then analyze the core dump with the appropriate debugger (gdb, delve, lldb) on a development machine.

Using kubectl run to Isolate Variables

Create a one-off pod with the same image but stripped of probes, ConfigMaps, and Secrets to isolate the failure:

# Run a bare pod to test if the image itself works
kubectl run test-api-gateway \
  --image=registry.example.com/api-gateway:v2.4.1 \
  --restart=Never \
  --rm -it \
  --command -- /bin/sh

# Inside the shell, manually run the application entrypoint
/app/server --config /tmp/test-config.yaml
# Observe if it crashes. If it runs fine, the issue is in your Kubernetes configuration.

Prevention: Best Practices to Avoid CrashLoopBackOff

1. Implement Proper Error Handling and Graceful Shutdown

Every application should handle SIGTERM gracefully. When Kubernetes decides to stop a pod (for rollout, node drain, or probe failure), it sends SIGTERM and waits for terminationGracePeriodSeconds (default 30 seconds) before sending SIGKILL. Your application must catch SIGTERM and shut down cleanly:

// Node.js example
process.on('SIGTERM', () => {
  console.log('Received SIGTERM, shutting down gracefully...');
  server.close(() => {
    console.log('HTTP server closed');
    db.disconnect().then(() => {
      console.log('Database connection closed');
      process.exit(0);  // Exit code 0 prevents CrashLoopBackOff from SIGTERM
    });
  });
  // Force exit after 25 seconds if clean shutdown fails
  setTimeout(() => {
    console.error('Forced shutdown after timeout');
    process.exit(1);
  }, 25000);
});

2. Distinguish Liveness, Readiness, and Startup Probes

• Startup probe: Protects slow-starting containers. Once it succeeds, it never runs again.
• Liveness probe: Determines if the container is still alive. Should be lightweight and dependency-free. Failure triggers a restart.
• Readiness probe: Determines if the container can serve traffic. Can check external dependencies. Failure removes the pod from service endpoints but does NOT restart it.

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  failureThreshold: 30    # 30 * 10s = 5 minutes max startup time
  periodSeconds: 10

livenessProbe:
  httpGet:
    path: /healthz         # Internal-only check
    port: 8080
  periodSeconds: 20
  failureThreshold: 3
  timeoutSeconds: 5

readinessProbe:
  httpGet:
    path: /ready           # Can check DB, cache, external services
    port: 8080
  periodSeconds: 10
  failureThreshold: 2
  timeoutSeconds: 3

3. Set Resource Requests and Limits Thoughtfully

• Always set memory requests equal to or slightly below typical usage to ensure the scheduler places pods on nodes with enough capacity.
• Set memory limits with headroom above peak usage (typically 20-50% above expected max).
• Use Vertical Pod Autoscaler (VPA) in recommendation mode to analyze historical usage patterns.

# Example: a Java application with -XX:MaxRAMPercentage=75.0
# If the container limit is 1Gi, the JVM will use at most 768Mi for heap
# Add 256Mi for native memory, metaspace, thread stacks → total ~1Gi
resources:
  requests:
    memory: "768Mi"
    cpu: "500m"
  limits:
    memory: "1280Mi"    # 1.25Gi — gives headroom above 1Gi
    cpu: "1000m"

4. Use Pod Disruption Budgets and Anti-Affinity

A single pod crash shouldn't take down your entire service. Spread replicas across nodes and availability zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - api-gateway
              topologyKey: kubernetes.io/hostname
      containers:
      - name: api-gateway
        image: registry.example.com/api-gateway:v2.4.1

5. Add Pre-stop Hooks for Clean Shutdown

Use preStop hooks to give your application time to drain in-flight requests before termination:

lifecycle:
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - |
        # Signal the application to stop accepting new connections
        curl -X POST http://localhost:8080/stop-accepting
        # Wait for existing connections to drain (up to 25 seconds)
        sleep 25
terminationGracePeriodSeconds: 30

6. Implement Structured Logging and Log-Level Control

When a crash occurs, you need maximum context. Use environment variables to control log verbosity without rebuilding the image:

env:
- name: LOG_LEVEL
  value: "debug"          # Set to "info" or "warn" in production normally
- name: LOG_FORMAT
  value: "json"           # Structured logs are easier to query

7. Monitor Restart Counts and Alert Proactively

Don't wait for a human to notice the CrashLoopBackOff status. Set up alerts based on restart count metrics:

# Prometheus alert rule for high restart rates
groups:
- name: kubernetes-pods
  rules:
  - alert: HighPodRestartRate
    expr: