Troubleshooting MetalLB Load Balancer: Common Issues and Fixes

MetalLB is a bare-metal load balancer implementation for Kubernetes that fills a critical gap: when you run Kubernetes on your own infrastructure (not on a cloud provider), the LoadBalancer service type has no native implementation. MetalLB provides one using standard networking protocols like ARP, BGP, or L2. However, because it interacts directly with your network fabric, misconfigurations and environmental issues are common. This tutorial walks you through diagnosing and fixing the most frequent MetalLB problems.

What Is MetalLB and Why It Matters

In managed Kubernetes offerings like EKS, GKE, or AKS, creating a Service of type LoadBalancer provisions a cloud load balancer automatically. On bare metal, that request silently hangs in a Pending state forever. MetalLB solves this by allocating IP addresses from a configured pool and announcing them to the network — either via Layer 2 (ARP/NDP) or BGP. Without MetalLB (or an equivalent), bare-metal clusters are limited to NodePort or ingress-based exposure, which are often insufficient for production workloads.

Prerequisites and Initial Diagnosis

Before diving into specific issues, ensure you have the right tools and a baseline understanding of your cluster state. You will need kubectl access and, ideally, access to the underlying nodes for network-level debugging.

Start by checking the overall health of MetalLB components:

kubectl get pods -n metallb-system
kubectl get svc -n metallb-system
kubectl logs -n metallb-system -l app.kubernetes.io/component=controller --tail=50
kubectl logs -n metallb-system -l app.kubernetes.io/component=speaker --tail=50

If any pod is not in a Running state, that is your first clue. Below, we cover the most common issues in detail.

Issue 1: Service Stuck in Pending State

The single most common symptom is a LoadBalancer service that never receives an external IP. The output of kubectl get svc shows Pending indefinitely.

First, inspect the service events:

kubectl describe svc my-loadbalancer-service

Look for messages like "Failed to allocate IP" or "no available IPs". These indicate that MetalLB's IP address pool is exhausted or misconfigured.

Verify your IPAddressPool configuration:

kubectl get ipaddresspool -n metallb-system -o yaml
kubectl get IPAddressPool -A

A correct pool definition looks like this:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool

Common causes and fixes for the Pending state include:

No IPAddressPool defined: Create one with a valid CIDR range that does not overlap with existing node IPs or DHCP leases.
Pool exhausted: All IPs are allocated. Expand the range or release unused services.
No L2Advertisement or BGPAdvertisement: The pool exists but MetalLB has no mechanism to announce it. Add the appropriate advertisement resource.
Namespace mismatch: MetalLB CRs must live in the metallb-system namespace in recent versions.

Issue 2: External IP Assigned but Unreachable

Sometimes MetalLB assigns an IP, but traffic to that IP times out or is refused. This is typically a Layer 2 or routing problem.

For L2 mode, MetalLB elects a single node to respond to ARP requests for the service IP. If that node is unreachable or the ARP announcement is being dropped, clients cannot connect.

Check which node is the leader for a given service:

kubectl logs -n metallb-system -l app.kubernetes.io/component=speaker | grep "192.168.1.240"

From a client machine on the same subnet, verify ARP resolution:

arping 192.168.1.240
ip neigh show | grep 192.168.1.240

If ARP fails, possible causes include:

The client is on a different subnet: L2 mode only works within the broadcast domain. For cross-subnet traffic, use BGP mode or a router with proxy-ARP enabled.
Switch ARP caching: Some switches cache stale ARP entries. Clear the cache or reduce the ARP cache timeout.
Network policy or firewall: Ensure the node's firewall allows ARP and the service traffic.
Memberlist secret misconfigured: The speaker pods use a secret for memberlist communication. If it is missing or invalid, leader election fails. Recreate it:

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Issue 3: BGP Sessions Not Establishing

In BGP mode, MetalLB speakers establish BGP peering sessions with upstream routers. If sessions do not come up, no routes are advertised and services are unreachable from outside the cluster.

Check the BGP peer status by examining speaker logs:

kubectl logs -n metallb-system -l app.kubernetes.io/component=speaker | grep -i bgp

Verify your BGPPeer resource:

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: my-router
  namespace: metallb-system
spec:
  myASN: 64500
  peerASN: 64501
  peerAddress: 192.168.1.1
  peerPort: 179

Common BGP issues include:

ASN mismatch: The myASN and peerASN must match what the router expects. A mismatch causes the session to be rejected.
Router not configured to peer: The upstream router must have a BGP neighbor entry pointing to each node's IP with the correct ASN.
Port 179 blocked: Firewalls often block BGP traffic. Verify connectivity with nc -zv 192.168.1.1 179 from a node.
Hold timer or keepalive mismatch: If timers differ significantly between peers, sessions flap. Align them in the BGPPeer spec using holdTime and keepaliveTime.

On the router side (example for FRR), verify the session:

vtysh -c "show bgp summary"
vtysh -c "show bgp neighbors"

Issue 4: Duplicate IP Address Conflicts

If MetalLB assigns an IP that is already in use elsewhere on the network, you will see intermittent connectivity, ARP conflicts, and MAC address flapping. This is one of the most damaging issues because it can disrupt other hosts.

Always reserve the MetalLB IP range outside of any DHCP pool. For example, if your DHCP server hands out 192.168.1.100-192.168.1.200, configure MetalLB to use 192.168.1.240-192.168.1.250.

To detect conflicts, run from a machine on the same subnet:

arping -D -I eth0 192.168.1.240
# A response means the IP is already in use

If a conflict is detected, immediately remove the conflicting service or change the pool:

kubectl edit ipaddresspool default-pool -n metallb-system
# Change the address range to an unused block

Then delete and recreate affected services so they pick up new IPs:

kubectl delete svc my-loadbalancer-service
kubectl apply -f my-service.yaml

Issue 5: Speaker Pods CrashLooping

If speaker pods are in CrashLoopBackOff, MetalLB cannot function at all. Inspect the logs:

kubectl logs -n metallb-system <speaker-pod-name> --previous

Frequent causes include:

Missing memberlist secret: Recreate as shown in Issue 2.
Invalid configuration CRs: A malformed IPAddressPool or BGPPeer can crash the speaker. Validate your YAML and check the controller logs for reconciliation errors.
RBAC issues: Ensure the MetalLB service accounts have the required permissions. Reapplying the manifest with the correct RBAC YAML usually resolves this.
Node network interface name mismatch: If you specified an interface in the config but the node uses a different name (e.g., eth0 vs ens192), the speaker may fail. Let MetalLB auto-detect by omitting the interface field.

Issue 6: Traffic Only Reaches One Node

In L2 mode, this is expected behavior: only the leader node receives traffic for a given service IP. MetalLB does not load balance across nodes in L2 mode; it relies on the leader election mechanism. If the leader node fails, failover occurs after a delay (typically a few seconds).

If you need true multi-node load balancing, switch to BGP mode where the router can use ECMP to distribute traffic across all nodes:

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: my-router
  namespace: metallb-system
spec:
  myASN: 64500
  peerASN: 64501
  peerAddress: 192.168.1.1
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: bgp-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.0.0.0/24
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgp-advert
  namespace: metallb-system
spec:
  ipAddressPools:
  - bgp-pool

Ensure your router supports ECMP and has it enabled for the BGP session.

Issue 7: Upgrading MetalLB Breaks Configuration

Major version upgrades of MetalLB often change the configuration API. Older versions used a ConfigMap named config in the metallb-system namespace, while newer versions (0.13+) use CRDs like IPAddressPool, L2Advertisement, and BGPPeer.

If you upgraded and services stopped working, check for leftover ConfigMap configurations:

kubectl get configmap config -n metallb-system

If it exists, migrate it to CRDs. MetalLB provides a manifest converter, but manual migration is straightforward. Remove the old ConfigMap after creating the equivalent CRs:

kubectl delete configmap config -n metallb-system

Always review the upgrade notes in the MetalLB release documentation before upgrading, and test in a staging cluster first.

Best Practices for Reliable MetalLB Deployments

Reserve IP ranges carefully: Coordinate with your network team to ensure MetalLB pools are excluded from DHCP and statically assigned elsewhere.
Use BGP for production: L2 mode is simpler but has single-node bottlenecks and slower failover. BGP with ECMP provides better scalability and redundancy.
Monitor speaker and controller pods: Set up alerts on pod restarts and log errors. A failing speaker can silently break service availability.
Keep MetalLB updated: Stay on a supported version and review changelogs for security and stability fixes.
Document your IP allocations: Maintain a record of which services use which external IPs to avoid conflicts during scaling.
Test failover regularly: In L2 mode, simulate node failure to confirm failover works within acceptable timeframes.
Use strict ARP in kube-proxy: If using ipvs mode in kube-proxy, ensure strictARP: true is set to prevent ARP conflicts with MetalLB:

kubectl edit configmap kube-proxy -n kube-system
# Set:
# mode: "ipvs"
# ipvs:
#   strictARP: true
kubectl rollout restart -n kube-system daemonset/kube-proxy

Conclusion

MetalLB is an essential tool for bare-metal Kubernetes clusters, but its tight coupling with the underlying network means troubleshooting requires both Kubernetes and networking knowledge. By systematically checking pod health, IP pool configuration, advertisement resources, and network-level behavior like ARP and BGP, you can quickly isolate and resolve most issues. Following best practices around IP allocation, mode selection, and monitoring will prevent many problems before they affect production traffic. Keep your configurations version-appropriate, test changes in staging, and maintain clear documentation of your IP assignments to ensure long-term reliability.