Introduction to Weave Networking
Weave Net is a popular software-defined networking (SDN) solution designed for containerized environments, particularly Kubernetes and Docker clusters. It creates a virtual network that connects application containers across multiple hosts, enabling seamless communication regardless of where the containers are physically located. Weave Net uses a lightweight overlay network built on top of the existing host network, encrypting traffic between nodes and providing built-in service discovery.
Despite its robust design, Weave Net can encounter issues related to IP allocation, peer connectivity, encryption, and performance. This tutorial walks through the most common problems developers and operators face when running Weave Net in production, along with practical fixes and diagnostic techniques.
Why Troubleshooting Weave Matters
Networking is the backbone of any distributed application. When Weave Net malfunctions, the symptoms can be subtle: intermittent timeouts, failed health checks, or pods stuck in ContainerCreating state. Left unresolved, these issues can cascade into application outages, data inconsistency, and degraded user experience. Understanding how to diagnose and fix Weave Net problems quickly is essential for maintaining reliable Kubernetes and Docker deployments.
Prerequisites and Setup
Before diving into troubleshooting, ensure you have the following tools available on your cluster nodes:
kubectlconfigured with cluster admin accessdockerorcontainerdCLI for container inspectionweavescript available on the host (typically at/usr/local/bin/weave)- SSH access to the affected nodes
- Basic familiarity with Linux networking commands (
ip,iptables,tcpdump)
You can verify that Weave is running on a node with the following command:
kubectl get pods -n kube-system -l name=weave-net
The output should show one Weave pod per node, all in the Running state. If any pod is in CrashLoopBackOff or Pending, you already have a starting point for investigation.
Common Issue 1: Peers Not Connecting
Symptoms
Nodes appear in the Weave peer list but show as unreachable. Cross-node pod communication fails while same-node communication works fine.
Diagnosis
Use the weave status command to inspect the peer topology:
# Execute inside the weave-net pod
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local status
# Or directly on the host
weave status
Look for the Connections section. Healthy peers should show a status of established. If you see connecting or failed, there is a connectivity problem between peers.
Fixes
The most common cause is firewall rules blocking Weave's default ports. Weave uses TCP port 6783 for control traffic and UDP port 6783 and 6784 for data traffic. Ensure these ports are open between all nodes:
# On Ubuntu/Debian with UFW
sudo ufw allow 6783/tcp
sudo ufw allow 6783/udp
sudo ufw allow 6784/udp
# On CentOS/RHEL with firewalld
sudo firewall-cmd --permanent --add-port=6783/tcp
sudo firewall-cmd --permanent --add-port=6783/udp
sudo firewall-cmd --permanent --add-port=6784/udp
sudo firewall-cmd --reload
If firewall rules are correct, verify that the peer addresses are reachable. Weave discovers peers through the IPALLOC_RANGE and initial peer list. You can manually test connectivity:
# Test TCP connectivity to a peer
nc -zv <peer-ip> 6783
# Check if the peer is listening
ss -tlnp | grep 6783
Another common cause is nodes with multiple network interfaces. Weave may bind to the wrong interface. You can specify the interface explicitly when launching Weave:
# Set the environment variable in the weave-net DaemonSet
env:
- name: WEAVE_IPALLOC_RANGE
value: "10.32.0.0/12"
- name: WEAVE_EXTRA_ARGS
value: "--iface=eth0"
Common Issue 2: IP Allocation Exhaustion
Symptoms
New pods fail to start with errors like Failed to create pod sandbox or no IP addresses available in range. The weave-npc or weave container logs show IP allocation failures.
Diagnosis
Check the IP allocation status:
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local status ipam
This command displays the IP range, the number of allocated IPs, and the number of free IPs. If the free count is zero or near zero, you have exhausted your IP range.
Fixes
The default IP allocation range for Weave is 10.32.0.0/12, which provides over one million IP addresses. However, if you customized this range or have a very large cluster, you may need to expand it. Update the Weave DaemonSet:
kubectl set env daemonset weave-net -n kube-system IPALLOC_RANGE=10.32.0.0/10
After changing the range, restart the Weave pods:
kubectl delete pods -n kube-system -l name=weave-net
If you suspect leaked IPs from deleted pods that were not properly released, you can force a reclaim. First, identify the orphaned IPs:
# List all allocated IPs
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local ps
# Compare with actual running pods
kubectl get pods --all-namespaces -o wide
To manually release an IP that is no longer in use:
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local detach <container-id>
Common Issue 3: Encrypted Traffic Overhead and Performance
Symptoms
Network throughput between pods on different nodes is significantly lower than expected. CPU usage on nodes is high, particularly in kernel processes related to encryption.
Diagnosis
Weave supports optional encryption using NaCl-based fast encryption. While secure, it adds CPU overhead. Check if encryption is enabled:
kubectl get daemonset weave-net -n kube-system -o yaml | grep -i password
If the WEAVE_PASSWORD environment variable is set, encryption is active. Measure the performance impact using a simple throughput test:
# On the server pod
iperf3 -s
# On the client pod
iperf3 -c <server-pod-ip> -t 30
Fixes
If encryption is not strictly required for your threat model, disabling it can significantly improve throughput. Remove the password environment variable:
kubectl set env daemonset weave-net -n kube-system WEAVE_PASSWORD-
If encryption is required, consider using a smaller MTU to reduce fragmentation. Weave's default MTU is 1376 for encrypted networks. You can tune this:
kubectl set env daemonset weave-net -n kube-system WEAVE_MTU=1400
Additionally, ensure that your nodes have sufficient CPU resources. Encryption is CPU-intensive, and resource-starved nodes will struggle. You can also enable fastdp (Fast Data Path), which uses the kernel's datapath for better performance:
# Verify fastdp status
weave status | grep "Datapath"
# If not using fastdp, check for kernel module issues
lsmod | grep openvswitch
modprobe openvswitch
Common Issue 4: DNS Resolution Failures
Symptoms
Pods can communicate via IP addresses but fail to resolve service names. CoreDNS or kube-dns appears healthy, but name resolution times out intermittently.
Diagnosis
Weave provides built-in DNS resolution for container names. However, in Kubernetes, DNS is typically handled by CoreDNS. Conflicts can arise when Weave's DNS intercepts queries. Test DNS resolution from within a pod:
kubectl exec -it <pod-name> -- nslookup kubernetes.default
# Test with a longer timeout
kubectl exec -it <pod-name> -- nslookup -timeout=10 kubernetes.default
Check the Weave DNS configuration:
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local status dns
Fixes
If Weave's DNS is interfering with CoreDNS, disable Weave's built-in DNS by passing the --no-dns flag:
kubectl set env daemonset weave-net -n kube-system WEAVE_EXTRA_ARGS="--no-dns"
Restart the Weave pods for the change to take effect:
kubectl delete pods -n kube-system -l name=weave-net
If the issue persists, verify that the CoreDNS pods are scheduled on nodes with healthy Weave networking:
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
Ensure that the CoreDNS service has the correct cluster IP and that iptables rules are not being overwritten by Weave's network policy controller:
# Check iptables rules related to DNS
sudo iptables -L -n -v | grep -i dns
# Verify the kube-dns service
kubectl get svc -n kube-system kube-dns
Common Issue 5: Network Policy Not Enforcing
Symptoms
Network policies defined in Kubernetes are not being enforced. Pods that should be isolated can communicate freely with restricted services.
Diagnosis
Weave Net includes a Network Policy Controller (NPC) that enforces Kubernetes NetworkPolicy objects. Check if the NPC container is running within the Weave pod:
kubectl get pods -n kube-system -l name=weave-net -o jsonpath='{.items[0].spec.containers[*].name}'
You should see both weave and weave-npc containers. If weave-npc is missing, the policy controller is not running.
Check the NPC container logs for errors:
kubectl logs -n kube-system <weave-pod-name> -c weave-npc
Fixes
If the NPC container is crashing, it may be due to insufficient RBAC permissions. Verify that the Weave service account has the necessary cluster role bindings:
kubectl get clusterrolebinding weave-net
kubectl get clusterrole weave-net -o yaml
If the role is missing or incomplete, re-apply the Weave manifest which includes the correct RBAC configuration:
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
If NPC is running but policies are still not enforced, check the iptables rules on the node. The NPC uses iptables to enforce policies:
# Look for WEAVE-NPC chains
sudo iptables -L -n -v | grep -i weave
# Check the specific NPC chain
sudo iptables -L WEAVE-NPC -n -v
If the chains are empty despite having network policies defined, restart the NPC container:
kubectl delete pod -n kube-system <weave-pod-name>
Common Issue 6: Stale Routes and Ghost Pods
Symptoms
Traffic is routed to non-existent pods. Connection attempts to a service result in timeouts even though the service and endpoints exist. The issue often appears after node failures or abrupt pod terminations.
Diagnosis
Weave maintains a routing table that maps pod IPs to their host nodes. When a node fails or a pod is deleted ungracefully, stale entries can remain. Inspect the routing table:
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local status routes
Look for routes pointing to IPs that no longer correspond to running pods. Cross-reference with the actual pod list:
kubectl get pods --all-namespaces -o wide | grep <suspected-ip>
Fixes
To flush stale routes, you can restart the Weave agent on the affected node. This forces it to re-synchronize its routing table with the rest of the cluster:
# Identify the weave pod on the affected node
kubectl get pods -n kube-system -l name=weave-net -o wide
# Delete it to trigger a restart
kubectl delete pod -n kube-system <weave-pod-on-affected-node>
For persistent stale entries, you may need to rm the Weave data store on the affected node. This is a more aggressive approach and should be done carefully:
# Stop weave
systemctl stop weave
# Remove the data store
rm -rf /var/lib/weave/weavedb.db
# Restart weave
systemctl start weave
After clearing the data store, the node will rejoin the cluster and rebuild its routing information from scratch.
Best Practices for Weave Net Operations
Monitor Weave Health Proactively
Do not wait for failures to investigate Weave. Set up monitoring for key metrics such as peer connection status, IP allocation usage, and packet drop rates. You can expose Weave metrics using the Prometheus format:
kubectl set env daemonset weave-net -n kube-system WEAVE_METRICS_ADDR=0.0.0.0:6782
Then scrape the metrics endpoint at http://<node-ip>:6782/metrics using Prometheus.
Keep Weave Updated
Weave Net receives regular updates that fix bugs and improve performance. Always run a supported version compatible with your Kubernetes version. Check the current version:
kubectl exec -n kube-system <weave-pod-name> -c weave -- /home/weave/weave --local version
Use Consistent MTU Across the Cluster
MTU mismatches cause packet fragmentation, which degrades performance and can lead to mysterious connectivity issues. Ensure all nodes use the same MTU for their primary network interface and that Weave's MTU is set to accommodate the overlay overhead:
# Check host interface MTU
ip link show eth0 | grep mtu
# Set Weave MTU accounting for 50 bytes of overlay overhead
kubectl set env daemonset weave-net -n kube-system WEAVE_MTU=1400
Document Your Network Topology
Maintain documentation of your Weave configuration, including the IP allocation range, peer list, encryption settings, and any custom flags. This dramatically speeds up troubleshooting when issues arise.
Test Network Policies in Staging
Network policy misconfigurations can silently block legitimate traffic. Always test policies in a staging environment before applying them to production. Use tools like kubectl-netshoot or a debug pod to verify connectivity:
kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash
# Inside the pod, test connectivity
curl -v http://<target-service>:<port>
Conclusion
Weave Net is a powerful and flexible networking solution for containerized environments, but like any complex distributed system, it requires careful operation and troubleshooting. By understanding the common issues covered in this tutorial — peer connectivity problems, IP exhaustion, encryption overhead, DNS conflicts, network policy enforcement failures, and stale routes — you can quickly diagnose and resolve problems before they impact your applications. The key to effective Weave troubleshooting is methodical diagnosis: start with weave status, examine logs, verify network-level connectivity, and apply targeted fixes. Combined with proactive monitoring and adherence to best practices, this approach will keep your Weave-powered cluster running smoothly and reliably.