← Back to DevBytes
Developer Tutorial August 13, 2026 8 min read

Troubleshooting Flannel Networking: Common Issues and Fixes

Introduction to Flannel Networking

Flannel is one of the most widely adopted Container Network Interface (CNI) plugins for Kubernetes. Originally developed by CoreOS, it provides a simple, layer 3 IPv4 network fabric designed to enable pod-to-pod communication across a cluster of nodes. Flannel works by assigning each node a unique subnet from a larger address space and then encapsulating or routing traffic between those subnets.

Despite its simplicity, Flannel can encounter a variety of networking issues that disrupt cluster communication. Understanding how to diagnose and resolve these problems is essential for any Kubernetes operator or platform engineer responsible for maintaining production clusters.

Why Flannel Troubleshooting Matters

When Flannel malfunctions, the symptoms can be severe: pods cannot reach each other, DNS resolution fails, services become unreachable, and applications break in confusing ways. Because networking sits at the foundation of Kubernetes, a single misconfiguration can cascade into cluster-wide outages. Quick, accurate diagnosis minimizes downtime and prevents engineers from chasing red herrings in application logs when the real problem lies in the network layer.

Understanding Flannel Architecture

Before diving into troubleshooting, it helps to understand how Flannel operates. Flannel runs as a DaemonSet on every node in the cluster. Each instance of the flanneld process reads its configuration from the Kubernetes API server and stores subnet assignments in etcd (or directly via the API server in modern versions). Flannel supports several backend encapsulation modes, each with different characteristics.

Flannel Backend Modes

The backend mode you choose affects which issues you are likely to encounter. VXLAN issues often involve MTU mismatches, while host-gw problems typically stem from routing table errors or missing layer 2 connectivity.

Common Flannel Issues and How to Diagnose Them

Issue 1: Pods Cannot Communicate Across Nodes

This is the most common Flannel symptom. Pods on the same node can talk to each other, but pods on different nodes cannot. The root cause is usually a problem with the flannel overlay, incorrect subnet allocation, or firewall rules blocking the VXLAN UDP port 8472.

Start your diagnosis by checking the flannel pods themselves:

kubectl get pods -n kube-flannel -o wide

If any flannel pod is in a CrashLoopBackOff or pending state, inspect its logs:

kubectl logs -n kube-flannel <flannel-pod-name>
kubectl describe pod -n kube-flannel <flannel-pod-name>

Next, verify that the flannel interface exists on each node. SSH into a node and run:

ip addr show flannel.1

If the interface is missing, flanneld may have failed to start or may be misconfigured. Check the flannel configuration stored in the cluster:

kubectl get configmap kube-flannel-cfg -n kube-flannel -o yaml

For firewall issues, ensure that UDP port 8472 is open between all nodes. On a system using iptables, you can verify with:

sudo iptables -L INPUT -n -v | grep 8472

If the port is blocked, add a rule to allow VXLAN traffic:

sudo iptables -I INPUT -p udp --dport 8472 -j ACCEPT

Issue 2: MTU Mismatches Causing Packet Loss

VXLAN encapsulation adds 50 bytes of overhead to each packet. If the underlying network has a standard MTU of 1500, the effective MTU for pod traffic becomes 1450. When Flannel does not account for this overhead, large packets get silently dropped, leading to symptoms like hanging connections, failed downloads, and intermittent connectivity.

Diagnose MTU issues by checking the flannel interface MTU on a node:

ip link show flannel.1 | grep mtu

Then test connectivity with packets of varying sizes. From inside a pod, run:

kubectl exec -it <pod-name> -- ping -M do -s 1472 <target-ip>
kubectl exec -it <pod-name> -- ping -M do -s 1422 <target-ip>

If the 1472-byte packet fails but the 1422-byte packet succeeds, you have an MTU problem. Fix it by setting the MTU explicitly in the flannel configuration. Edit the kube-flannel-cfg ConfigMap:

kubectl edit configmap kube-flannel-cfg -n kube-flannel

Add or modify the MTU field in the net-conf.json section:

{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "vxlan",
    "MTU": 1450
  }
}

After saving, restart the flannel pods to apply the change:

kubectl delete pods -n kube-flannel -l app=flannel

Issue 3: Incorrect Subnet Allocation

Flannel assigns each node a subnet from the cluster CIDR. If subnet allocation becomes inconsistent, for example after nodes are removed and re-added, you can end up with overlapping subnets or nodes missing their assignments. This causes routing conflicts and dropped traffic.

Check the subnet assigned to each node by examining the node annotations:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.flannel\.alpha\.coreos\.com\/backend-data}{"\n"}{end}'

You can also inspect the flannel subnet file on each node directly:

cat /run/flannel/subnet.env

A healthy subnet.env file looks like this:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

If the subnet information is missing or incorrect, delete the subnet file and restart flannel to force re-registration:

sudo rm /run/flannel/subnet.env
sudo systemctl restart flanneld

In Kubernetes-managed Flannel deployments, restart the flannel pod instead:

kubectl delete pod -n kube-flannel <flannel-pod-name>

Issue 4: CNI Configuration Problems

Flannel relies on a CNI configuration file located at /etc/cni/net.d/ on each node. If this file is missing, corrupted, or references the wrong bridge or subnet, pods will fail to obtain IP addresses or will be placed on the wrong network.

Check the CNI configuration on a node:

cat /etc/cni/net.d/10-flannel.conflist

A correct configuration should look similar to this:

{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

If the file is missing, you can recreate it manually or reapply the Flannel DaemonSet, which typically restores the configuration. Ensure the kubelet on the node has permissions to read this directory:

ls -la /etc/cni/net.d/

Issue 5: CoreDNS and Service Discovery Failures

Flannel problems often manifest as DNS failures because DNS queries traverse the pod network. If CoreDNS pods cannot reach upstream resolvers or if pods cannot reach CoreDNS, applications fail to resolve service names.

First, verify that CoreDNS is running:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

Test DNS resolution from inside a pod:

kubectl exec -it <pod-name> -- nslookup kubernetes.default.svc.cluster.local

If DNS fails, check whether the pod can reach the CoreDNS service IP:

kubectl exec -it <pod-name> -- ping <coredns-service-ip>

If the ping fails but pod-to-pod communication works, the issue may be with kube-proxy or iptables rules rather than Flannel itself. Check kube-proxy logs:

kubectl logs -n kube-system <kube-proxy-pod-name>

If pod-to-pod communication also fails, return to the Flannel troubleshooting steps above, as the DNS issue is a symptom of the underlying network problem.

Advanced Diagnostic Techniques

Using tcpdump to Capture Flannel Traffic

When logical checks do not reveal the problem, packet captures provide definitive evidence. Use tcpdump on the physical interface to verify VXLAN packets are flowing between nodes:

sudo tcpdump -i eth0 udp port 8472 -nn -c 20

If you see no packets when traffic should be flowing, the issue is likely upstream firewalls or routing. Capture on the flannel interface to verify decapsulation is working:

sudo tcpdump -i flannel.1 -nn -c 20

Inspecting the Flannel etcd Data

In clusters where Flannel uses etcd directly (older deployments), you can inspect subnet leases:

ETCDCTL_API=2 etcdctl --endpoints=https://<etcd-endpoint>:2379 \
  --ca-file=/etc/etcd/ca.pem \
  --cert-file=/etc/etcd/client.pem \
  --key-file=/etc/etcd/client-key.pem \
  ls /coreos.com/network/subnets

Stale subnet leases from removed nodes can cause conflicts. Remove them with:

ETCDCTL_API=2 etcdctl --endpoints=https://<etcd-endpoint>:2379 \
  --ca-file=/etc/etcd/ca.pem \
  --cert-file=/etc/etcd/client.pem \
  --key-file=/etc/etcd/client-key.pem \
  rm /coreos.com/network/subnets/<stale-lease>

Verifying iptables Rules Created by Flannel

Flannel creates iptables rules for masquerading and traffic filtering. Inspect the relevant chains:

sudo iptables -t nat -L FLANNEL-POSTRTG -n -v
sudo iptables -t nat -L FLANNEL-PREROUTING -n -v

If these chains are missing, flanneld may not have started correctly or may lack the required capabilities. Ensure the Flannel DaemonSet grants the NET_ADMIN capability:

kubectl get ds kube-flannel-ds -n kube-flannel -o yaml | grep -A 5 securityContext

Best Practices for Flannel Reliability

Reinstalling Flannel as a Last Resort

When configuration fixes do not resolve the issue, a clean reinstallation often helps. Remove the existing Flannel resources and reinstall from a known-good manifest:

kubectl delete ds kube-flannel-ds -n kube-flannel
kubectl delete ns kube-flannel

Clean up residual interfaces and configuration files on each node:

sudo ip link delete flannel.1
sudo ip link delete cni0
sudo rm -rf /etc/cni/net.d/10-flannel.conflist
sudo rm -rf /run/flannel

Then reapply the Flannel manifest:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Wait for all flannel pods to become ready:

kubectl get pods -n kube-flannel -w

Verify cross-node pod communication before declaring the cluster healthy.

Conclusion

Flannel is a robust and straightforward CNI plugin, but its simplicity does not make it immune to networking problems. Most Flannel issues fall into a handful of categories: firewall restrictions blocking encapsulation traffic, MTU mismatches causing silent packet loss, subnet allocation conflicts, CNI configuration errors, and DNS failures that are symptoms of deeper network problems. By systematically checking flannel pod health, inspecting interfaces and routes, verifying firewall rules, and using tools like tcpdump for definitive evidence, you can quickly isolate and resolve the root cause. Following best practices such as pinning versions, setting MTU explicitly, and monitoring pod health will prevent many issues from occurring in the first place. With a solid troubleshooting methodology and an understanding of Flannel's architecture, you can keep your Kubernetes cluster network running reliably even under demanding production conditions.

— Ad —

Google AdSense will appear here after approval

← Back to all articles