Troubleshooting eBPF for Networking: Common Issues and Fixes

Troubleshooting eBPF for Networking: Common Issues and Fixes

Extended Berkeley Packet Filter (eBPF) has become a cornerstone of modern networking observability and datapath manipulation. From Cilium to Katran, eBPF powers load balancers, firewalls, and deep network observability tools. However, when eBPF programs misbehave in production, debugging can be notoriously difficult because the code runs in kernel context, interacts with hardware offloads, and competes with other kernel subsystems. This tutorial walks you through the most common eBPF networking issues and provides practical fixes you can apply today.

What Is eBPF for Networking?

eBPF is a sandboxed virtual machine inside the Linux kernel that allows you to run bytecode attached to various hook points such as xdp, tc, kprobe, and cgroup/skb. In networking, eBPF programs typically attach to XDP (eXpress Data Path) for early packet processing or to Traffic Control (tc) hooks for more complex classification and manipulation. The verifier ensures safety, but it cannot catch logical bugs, performance regressions, or interactions with other subsystems.

Why Troubleshooting Matters

A misconfigured eBPF program can silently drop packets, cause CPU spikes, or trigger verifier rejections that prevent your service from starting. In production environments running Kubernetes with CNI plugins like Cilium, a single faulty eBPF program can take down node-to-node communication. Understanding the failure modes and having a systematic debugging workflow is essential for any platform engineer working with eBPF-based networking.

Setting Up Your Debugging Environment

Before diving into specific issues, ensure you have the right tooling. You will need bpftool, llvm with BPF backend support, libbpf, and ideally bpftrace for live tracing. Install them with:

sudo apt-get install -y bpfcc-tools linux-tools-$(uname -r) \
  llvm clang libbpf-dev bpftool bpftrace

Verify your kernel supports BPF and check the relevant features:

bpftool feature probe | grep -E "eBPF|XDP|prog_type"
uname -r
cat /proc/sys/net/core/bpf_jit_enable

If bpf_jit_enable is set to 0, enable it for production performance:

echo 1 | sudo tee /proc/sys/net/core/bpf_jit_enable

Common Issue 1: Verifier Rejections

The verifier is the most common source of frustration. It rejects programs that exceed the instruction limit, access out-of-bounds memory, or have unreachable states. A typical rejection looks like:

bpf_load_program() err=13
0: (bf) r1 = r6
1: (61) r2 = *(u32 *)(r1 +0)
2: (07) r2 += 100
3: (61) r3 = *(u32 *)(r2 +0)
invalid bpf_context access off=100 size=4

This error means the program tried to read past the valid context structure. For XDP, the context is struct xdp_md, which only has three fields: data, data_end, and data_meta. The fix is to use proper bounds checking before any pointer dereference:

SEC("xdp")
int xdp_parse(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data     = (void *)(long)ctx->data;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_DROP;

    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *iph = (void *)(eth + 1);
    if ((void *)(iph + 1) > data_end)
        return XDP_DROP;

    return XDP_PASS;
}

Key rules to satisfy the verifier:

• Always check data + offset <= data_end before reading.
• Use barrier_var() when the compiler optimizes away checks the verifier needs.
• Keep programs under the 1 million instruction limit (older kernels cap at 4096).
• Avoid unbounded loops; use bpf_loop() or bounded #pragma unroll.

Common Issue 2: Maps Not Updating or Returning Stale Data

eBPF maps are the shared memory interface between kernel and userspace. A frequent bug is reading or writing maps with mismatched key sizes or expecting atomic updates that never happen. Symptoms include counters that never increment or stale routing decisions.

First, inspect loaded maps with bpftool:

sudo bpftool map show
sudo bpftool map dump id 42

If the map appears empty but your program is clearly running, the issue is often a mismatched key size. For example, declaring a map with __u32 keys in BPF but using int in userspace can cause silent failures on some architectures. Always use explicit fixed-width types:

struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 10000);
    __type(key, __u32);
    __type(value, struct flow_stats);
} flow_map SEC(".maps");

For LRU maps, remember that entries can be evicted at any time. If you need guaranteed persistence, use BPF_MAP_TYPE_HASH and manage eviction manually. Also, use bpf_map_lookup_and_delete_elem for queue-style semantics to avoid race conditions between lookup and delete.

Common Issue 3: Packets Silently Dropped

One of the most insidious problems is packets disappearing with no logs. XDP programs that return XDP_DROP do not generate logs by default. To diagnose, attach a tracepoint that counts drops and correlates them with your program:

bpftrace -e 'tracepoint:xdp:xdp_exception { @[args->act] = count(); }'

You can also use xdpdump from the xdp-tools project to capture packets at the XDP layer:

sudo xdpdump -i eth0 --rx-capture entry,exit

Common causes of silent drops include:

• Returning XDP_DROP on packets you intended to pass due to a logic error in protocol parsing.
• VLAN-tagged packets where eth->h_proto is ETH_P_8021Q instead of ETH_P_IP.
• Checksum offload interactions where the NIC has not yet computed checksums by the time XDP runs.
• Hardware offload conflicts when XDP is attached in XDP_FLAGS_HW_MODE but the driver does not support it.

A robust parser should handle VLAN tags explicitly:

struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end) return XDP_DROP;

__u16 h_proto = eth->h_proto;
void *next = (void *)(eth + 1);

// Handle single VLAN
if (h_proto == bpf_htons(ETH_P_8021Q)) {
    struct vlan_hdr *vlan = next;
    if ((void *)(vlan + 1) > data_end) return XDP_DROP;
    h_proto = vlan->h_vlan_encapsulated_proto;
    next = (void *)(vlan + 1);
}

if (h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS;

Common Issue 4: High CPU Usage and Lock Contention

eBPF programs run in softirq context, and expensive operations can starve other network processing. If you notice high SI CPU usage in top, profile your BPF program with perf:

sudo perf record -a -g -- sleep 10
sudo perf report | grep -i bpf

Common performance pitfalls and fixes:

• Per-CPU maps not used for counters: Always use BPF_MAP_TYPE_PERCPU_ARRAY or BPF_MAP_TYPE_PERCPU_HASH for statistics to avoid cache-line bouncing.
• Excessive map lookups in hot path: Cache lookups in registers and minimize hash map traversals.
• String comparisons in BPF: Use integer keys instead of comparing byte arrays.
• Unbounded tail calls: Limit bpf_tail_call() chaining to avoid stack overhead.

Convert a shared counter to a per-CPU counter like this:

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __type(key, __u32);
    __type(value, __u64);
    __uint(max_entries, 1);
} pkt_count SEC(".maps");

SEC("xdp")
int count_packets(struct xdp_md *ctx) {
    __u32 key = 0;
    __u64 *cnt = bpf_map_lookup_elem(&pkt_count, &key);
    if (cnt) __sync_fetch_and_add(cnt, 1);
    return XDP_PASS;
}

Common Issue 5: Program Fails to Attach

Attachment failures often produce cryptic errors like EINVAL or EBUSY. Common causes include:

• Another program is already attached to the same hook with conflicting flags.
• The interface is in promiscuous mode or has a conflicting tc qdisc.
• The driver does not support native XDP (use XDP_FLAGS_SKB_MODE as fallback).
• SELinux or LSM policies block BPF operations.

Diagnose with:

sudo bpftool net show dev eth0
sudo dmesg | grep -i bpf
sudo strace -e bpf ./your_loader eth0

To detach a stale program before reattaching:

sudo ip link set dev eth0 xdp off
sudo ip link set dev eth0 xdp obj prog.o sec xdp

If native XDP is unsupported, fall back to generic mode:

sudo ip link set dev eth0 xdpdrv off
sudo ip link set dev eth0 xdpgeneric obj prog.o sec xdp

Common Issue 6: CO-RE Compatibility Issues

Compile Once - Run Everywhere (CO-RE) relies on BTF (BPF Type Format) information. If your program works on one kernel but fails on another with errors like invalid argument: BTF, the target kernel may lack BTF. Check with:

ls /sys/kernel/btf/vmlinux
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

If BTF is missing, either enable CONFIG_DEBUG_INFO_BTF=y in the kernel config or use vmlinux.h generated from a compatible kernel. For field offsets that vary across kernel versions, use bpf_core_read() instead of direct dereferences:

struct task_struct *task = (void *)bpf_get_current_task();
__u32 pid;
bpf_core_read(&pid, sizeof(pid), &task->pid);

Best Practices for Production eBPF Networking

• Always bounds-check pointers before dereferencing, even when you are confident of packet structure.
• Use per-CPU maps for any write-heavy data to avoid contention.
• Log decisions with tracepipes using bpf_trace_printk() during development, then remove them in production.
• Pin maps with bpf_obj_pin() so userspace daemons can reconnect after restarts without losing state.
• Version your programs and include a magic number in map values to detect schema mismatches.
• Test with different MTUs and jumbo frames, as XDP sees the full frame before segmentation.
• Monitor verifier logs in CI by capturing stderr from bpftool prog load to catch regressions early.
• Use libbpf's skeleton for cleaner loader code and automatic map and program management.

Conclusion

Troubleshooting eBPF networking programs requires a methodical approach that combines kernel-level understanding with practical tooling. By mastering the verifier's constraints, using per-CPU maps correctly, handling VLAN and offload edge cases, and leveraging tools like bpftool, bpftrace, and xdpdump, you can quickly isolate and fix the most common failure modes. The key is to treat eBPF programs with the same rigor as any production code: write defensive parsers, test across kernel versions, monitor verifier output in CI, and always have a rollback path when deploying changes to the datapath. With these practices in place, eBPF becomes a reliable and powerful foundation for high-performance networking rather than a source of mysterious packet drops.