← Back to DevBytes
Developer Tutorial August 13, 2026 10 min read

Troubleshooting IPVS Load Balancing: Common Issues and Fixes

Introduction to IPVS Load Balancing

IPVS (IP Virtual Server) is a high-performance load balancer built into the Linux kernel. Operating at Layer 4 of the OSI model, it distributes incoming network traffic across multiple backend servers, providing scalability and high availability for services. Unlike user-space load balancers such as HAProxy or NGINX, IPVS runs directly in the kernel, making it significantly faster and more efficient for handling large volumes of traffic.

IPVS is the backbone of many production systems, including Kubernetes kube-proxy in IPVS mode, where it handles service routing across cluster nodes. Because of its critical role, when IPVS malfunctions, entire applications can become unreachable. Understanding how to diagnose and fix common IPVS issues is an essential skill for any infrastructure or platform engineer.

Why Troubleshooting IPVS Matters

When IPVS fails or behaves unexpectedly, the symptoms can be subtle: intermittent connection failures, uneven traffic distribution, or complete service outages. Because IPVS operates below the application layer, traditional application-level monitoring tools often cannot detect the root cause. This makes proactive troubleshooting crucial.

Common scenarios where IPVS troubleshooting becomes necessary include:

Understanding IPVS Architecture

Before diving into troubleshooting, it helps to understand how IPVS works. IPVS uses Netfilter hooks to intercept incoming packets destined for a virtual IP address (VIP). It then applies a scheduling algorithm to select a real server (backend) and rewrites the packet headers to forward traffic accordingly.

IPVS supports several forwarding methods:

Essential Tools for IPVS Troubleshooting

The primary tool for inspecting and managing IPVS is ipvsadm. If it is not installed, you can add it with your package manager:

# Debian/Ubuntu
sudo apt-get install ipvsadm

# RHEL/CentOS
sudo yum install ipvsadm

To view the current IPVS routing table:

sudo ipvsadm -L -n

The output will display each virtual service, its scheduling algorithm, and the associated real servers with their weights and connection statistics. Adding the --stats flag shows traffic counters, while --rate displays connection rates:

sudo ipvsadm -L -n --stats
sudo ipvsadm -L -n --rate

To verify that the IPVS kernel module is loaded:

lsmod | grep ip_vs

If the module is not loaded, you can manually load it:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh

Common Issue 1: IPVS Rules Not Persisting After Reboot

One of the most frequent complaints is that IPVS configuration disappears after a system restart. By default, IPVS rules exist only in memory and are lost on reboot. The solution is to save and restore the rules using ipvsadm-save and ipvsadm-restore.

To save the current rules to a file:

sudo ipvsadm-save > /etc/ipvsadm.rules

To restore rules from the file:

sudo ipvsadm-restore < /etc/ipvsadm.rules

On systemd-based systems, enable the ipvsadm service to automatically restore rules on boot:

sudo systemctl enable ipvsadm
sudo systemctl start ipvsadm

Common Issue 2: Uneven Traffic Distribution

If traffic is not being distributed evenly across backend servers, the first step is to check the scheduling algorithm in use. Different algorithms have different behaviors:

Inspect the current configuration:

sudo ipvsadm -L -n

If weights are incorrect, you can update them. For example, to set a weight of 3 for a specific real server:

sudo ipvsadm -e -t 192.168.1.100:80 -r 192.168.1.11:80 -w 3 -g

Another common cause of uneven distribution is connection persistence. If persistence is enabled, IPVS will send all connections from the same client to the same backend for a configured timeout period. Check for persistence with:

sudo ipvsadm -L -n -p

If persistence is causing problems, you can remove it by editing the service:

sudo ipvsadm -e -t 192.168.1.100:80 -s wlc

Note that omitting the -p flag when editing removes persistence. Be cautious with this in production, as some applications (like those with sticky sessions) may require persistence.

Common Issue 3: Backends Marked as Unreachable

If IPVS shows backends but no traffic is reaching them, the issue may be at the network layer. First, verify that the backend servers are actually reachable from the IPVS host:

ping 192.168.1.11
curl -v http://192.168.1.11:80/

Check whether the backend port is listening:

ssh user@192.168.1.11 "ss -tlnp | grep :80"

If the backend is reachable but IPVS is still not forwarding traffic, examine the IPVS connection table for dropped or expired connections:

sudo ipvsadm -L -n --connection

Look for connections stuck in SYN_RECV or FIN_WAIT states, which can indicate network issues between the IPVS host and backends.

Also verify that IP forwarding is enabled on the IPVS host, especially when using NAT mode:

cat /proc/sys/net/ipv4/ip_forward

If the output is 0, enable it:

sudo sysctl -w net.ipv4.ip_forward=1

To make this change persistent across reboots, add it to /etc/sysctl.conf:

echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Common Issue 4: Direct Routing (DR) Mode Not Working

DR mode is efficient but requires specific network configuration on both the IPVS host and the backend servers. The most common issue is that the backend servers do not have the VIP configured on their loopback interface.

On each backend server, configure the VIP on the loopback interface:

sudo ip addr add 192.168.1.100/32 dev lo

Additionally, the backend must suppress ARP responses for the VIP to prevent ARP conflicts. On each backend, set the following kernel parameters:

sudo sysctl -w net.ipv4.conf.lo.arp_ignore=1
sudo sysctl -w net.ipv4.conf.lo.arp_announce=2
sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2

To verify DR mode is configured correctly on the IPVS host, check that the forwarding method is set to gateway (-g):

sudo ipvsadm -L -n

The output should show Route under the Forward column for each real server, indicating DR mode. If it shows Masq, the server is in NAT mode and needs to be reconfigured:

sudo ipvsadm -e -t 192.168.1.100:80 -r 192.168.1.11:80 -g

Common Issue 5: IPVS in Kubernetes Not Routing Traffic

When running Kubernetes with kube-proxy in IPVS mode, service traffic issues often stem from misconfiguration. First, verify that kube-proxy is actually using IPVS:

kubectl logs -n kube-system -l k8s-app=kube-proxy | grep "Using ipvs"

Check the kube-proxy configuration:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

The mode should be set to ipvs. If it is not, update the ConfigMap and restart kube-proxy:

kubectl edit configmap kube-proxy -n kube-system
# Set mode: "ipvs"

kubectl delete pod -l k8s-app=kube-proxy -n kube-system

Verify that the IPVS rules are being created by kube-proxy:

sudo ipvsadm -L -n | grep -i "cluster"

If IPVS rules are missing, check that the required kernel modules are loaded on the node:

lsmod | grep -e ip_vs -e nf_conntrack

If modules are missing, load them and ensure they load on boot by creating a configuration file:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

Common Issue 6: Connection Timeout and Conntrack Table Exhaustion

IPVS relies on connection tracking (conntrack) for NAT mode. Under high traffic loads, the conntrack table can fill up, causing new connections to be dropped. Check the current conntrack usage:

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

If the count is close to the max, you need to increase the table size:

sudo sysctl -w net.netfilter.nf_conntrack_max=1048576

Also consider tuning conntrack timeout values to free up entries faster:

sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=54000
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

Make these persistent by adding them to /etc/sysctl.d/99-ipvs.conf:

cat <<EOF | sudo tee /etc/sysctl.d/99-ipvs.conf
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 54000
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.ipv4.ip_forward = 1
EOF

sudo sysctl -p /etc/sysctl.d/99-ipvs.conf

Common Issue 7: Health Checks Not Removing Failed Backends

IPVS itself does not perform health checks. It relies on external tools like keepalived or custom scripts to add and remove real servers. If a backend fails but remains in the IPVS table, traffic will continue to be sent to it.

If you are using keepalived, check its configuration and logs:

sudo systemctl status keepalived
sudo journalctl -u keepalived -f

A basic keepalived configuration with health checks looks like this:

virtual_server 192.168.1.100 80 {
    delay_loop 10
    lb_algo wrr
    lb_kind NAT
    protocol TCP

    real_server 192.168.1.11 80 {
        weight 3
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            retry 3
            delay_before_retry 2
        }
    }

    real_server 192.168.1.12 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 3
            retry 3
            delay_before_retry 2
        }
    }
}

If you are not using keepalived, you can implement a simple health check script that removes failed backends:

#!/bin/bash
VIP="192.168.1.100"
PORT=80
BACKENDS=("192.168.1.11" "192.168.1.12")

for backend in "${BACKENDS[@]}"; do
    if curl -s --connect-timeout 2 "http://${backend}:${PORT}/" > /dev/null; then
        # Backend is healthy, ensure it is in the table
        if ! ipvsadm -L -n | grep -q "${backend}:${PORT}"; then
            ipvsadm -a -t ${VIP}:${PORT} -r ${backend}:${PORT} -m
        fi
    else
        # Backend is down, remove it
        ipvsadm -d -t ${VIP}:${PORT} -r ${backend}:${PORT} 2>/dev/null
    fi
done

Schedule this script with cron to run at regular intervals:

*/10 * * * * /usr/local/bin/ipvs-health-check.sh

Debugging with Packet Captures

When IPVS issues are not obvious from configuration inspection, packet captures can reveal where traffic is being dropped. Use tcpdump on the IPVS host to trace incoming and outgoing packets:

# Capture traffic on the VIP
sudo tcpdump -i eth0 -n host 192.168.1.100 and port 80

# Capture traffic to a specific backend
sudo tcpdump -i eth0 -n host 192.168.1.11 and port 80

If you see packets arriving at the VIP but not being forwarded to backends, the issue is likely in the IPVS configuration or kernel routing. If packets are being forwarded but no response returns, the problem is on the backend or the return path.

For NAT mode, verify that return traffic from backends is routing through the IPVS host. The backend's default gateway must be the IPVS host's internal IP for NAT to work correctly.

Best Practices for IPVS in Production

Conclusion

IPVS is a powerful and efficient load balancer, but its kernel-level operation and tight coupling with network configuration mean that troubleshooting requires a systematic approach. By mastering tools like ipvsadm, understanding the differences between forwarding modes, and implementing proper health checking and persistence mechanisms, you can quickly diagnose and resolve most IPVS issues. The key is to methodically verify each layer of the stack — from kernel modules and sysctl parameters to IPVS rules, network connectivity, and backend health — until the root cause is identified. With the practices and techniques covered in this tutorial, you are well-equipped to keep your IPVS-based load balancing infrastructure running smoothly in production.

— Ad —

Google AdSense will appear here after approval

← Back to all articles