Troubleshooting Network Policies in Kubernetes: Common Issues and Fixes

Kubernetes Network Policies are a powerful mechanism for controlling pod-to-pod communication within a cluster. However, when they don't behave as expected, debugging can be frustrating because the symptoms—connection timeouts, refused connections, or partial reachability—often look identical to application-level failures. This tutorial walks you through the most common Network Policy issues, how to diagnose them, and how to fix them.

What Are Network Policies?

Network Policies are Kubernetes resources that specify how a group of pods are allowed to communicate with each other and with other network endpoints. They operate at the IP address or port level (OSI layers 3 and 4) and are enforced by a Container Network Interface (CNI) plugin that supports them, such as Calico, Cilium, Weave Net, or Antrea.

A Network Policy uses labels to select pods and defines rules that describe which traffic is allowed to or from those selected pods. Once a pod is selected by a policy, all traffic that is not explicitly allowed by any matching rule is denied.

Why Network Policies Matter

In a default Kubernetes cluster, all pods can communicate with all other pods without restriction. This "flat network" model is convenient for development but dangerous in production. Network Policies let you implement least-privilege segmentation, reducing the blast radius of a compromised pod and helping satisfy compliance requirements such as PCI-DSS or HIPAA.

However, because Network Policies are additive and enforced by the CNI, misconfigurations can silently break applications or, worse, give a false sense of security. Understanding how to troubleshoot them is essential for any platform or DevOps engineer running Kubernetes in production.

Prerequisites for Network Policy Enforcement

Before troubleshooting, confirm that your cluster actually enforces Network Policies. Not all CNIs support them. For example, the default flannel CNI does not implement NetworkPolicy. If you apply a policy and nothing changes, the CNI might be silently ignoring it.

# Check which CNI is in use
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get pods -n kube-system -l k8s-app=cilium

# Verify the CNI config on a node
cat /etc/cni/net.d/*.conflist

If your CNI does not support Network Policies, you must install a policy-enforcing plugin or switch CNIs. Calico and Cilium are the most common choices and can often run alongside an existing CNI in "policy-only" mode.

Issue 1: Default Deny Blocking All Traffic

The most common mistake is applying a default-deny policy without an accompanying allow policy. A default-deny ingress policy isolates the selected pods from everything, including legitimate clients.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

This policy selects all pods in the production namespace and denies all inbound traffic. If you apply this without a corresponding allow rule, your services become unreachable.

Fix: Always pair a default-deny with explicit allow policies. For example, allow ingress from pods with a specific label:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Issue 2: Label Selector Mismatches

Network Policies rely on labels to match pods. A typo or a missing label means the policy selects the wrong pods—or no pods at all. This is especially common when teams rename labels or use inconsistent naming conventions.

Diagnosis: Use kubectl get pods --show-labels to verify that the labels on your pods match the selectors in your policies.

# Show labels on all pods in a namespace
kubectl get pods -n production --show-labels

# Check which pods a selector matches
kubectl get pods -n production -l app=backend

Fix: Standardize labels across your organization and validate them in CI. Use a tool like kyverno or OPA Gatekeeper to enforce label conventions on admission.

Issue 3: Namespace Selectors Not Matching

When a Network Policy references another namespace using namespaceSelector, the selector matches labels on the Namespace resource, not on pods. A frequent mistake is assuming the namespace name itself is matched.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-dev
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              environment: dev

This policy allows traffic from namespaces labeled environment: dev. If the dev namespace does not have that label, no traffic will be allowed.

Diagnosis: Check namespace labels directly.

kubectl get namespace dev --show-labels

# Add the required label if missing
kubectl label namespace dev environment=dev

Issue 4: Forgetting to Allow Egress Traffic

Many teams focus on ingress and forget egress. When you apply a default-deny egress policy, pods cannot reach DNS, the Kubernetes API, external databases, or even other pods unless explicitly allowed.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress

After applying this, pods in the namespace lose outbound connectivity entirely. The most visible symptom is DNS resolution failure.

Fix: Explicitly allow DNS traffic to the kube-system namespace, where CoreDNS typically runs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Note that the label kubernetes.io/metadata.name is automatically added to namespaces in Kubernetes 1.21+, making namespace selectors more reliable.

Issue 5: Port and Protocol Mismatches

Network Policies match traffic on specific ports and protocols. If your application listens on port 8080 but your policy allows port 80, traffic is denied. Similarly, allowing only TCP when the application uses UDP causes silent failures.

Diagnosis: Compare the ports in your policy with the actual listening ports of your pods.

# Exec into a pod and check listening ports
kubectl exec -it backend-xxx -n production -- sh
netstat -tlnp

# Or check the container's exposed ports
kubectl get pod backend-xxx -n production -o jsonpath='{.spec.containers[*].ports}'

Fix: Ensure the ports section of your ingress and egress rules matches the actual application ports. If your service uses a different port than the container, remember that Network Policies operate on the pod's port, not the Service port.

Issue 6: Policies Are Additive and Order-Independent

A common misconception is that Network Policies are evaluated in order, like firewall rules. They are not. Policies are additive: if any policy allows a particular flow, it is permitted, regardless of other policies that might deny it. There is no explicit "deny" rule in the Kubernetes NetworkPolicy API.

This means that if you have a permissive policy somewhere in the namespace, your restrictive policies are effectively bypassed for the flows that the permissive policy allows.

Diagnosis: List all policies affecting a namespace and review them together.

kubectl get networkpolicy -n production
kubectl describe networkpolicy -n production

Fix: Audit all policies in a namespace holistically. Remove or tighten overly permissive policies. Consider using a policy management tool like Calico's hierarchical policies or Cilium's clusterwide policies for layered enforcement.

Issue 7: Traffic From Outside the Cluster

Network Policies apply to pod-to-pod traffic within the cluster. Traffic entering from external sources (via Ingress controllers, LoadBalancer services, or NodePort) often bypasses pod-level Network Policies because the source IP may be the node's IP or a SNAT'd address.

Diagnosis: Check whether external traffic is being source-NAT'd. If the source IP is the node IP, your podSelector and namespaceSelector rules will not match.

# Check the source IP seen by a pod
kubectl exec -it backend-xxx -n production -- sh
tcpdump -i eth0 -nn port 8080

Fix: To allow external traffic, include an ipBlock rule for the external CIDR, or allow traffic from the namespace where the Ingress controller runs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

Debugging Tools and Techniques

Beyond inspecting YAML, several tools help diagnose Network Policy issues in real time.

kubectl-trace: Run bpftrace programs on nodes to inspect packet flows.
Cilium Hubble: Provides flow visibility and policy verdicts for Cilium-managed clusters.
Calicoctl: Inspect Calico-specific policy state and endpoint details.
Temporary debug pods: Spin up a pod with networking tools like curl, nc, and tcpdump to test connectivity.

Here is a quick way to test connectivity from a debug pod:

# Launch a debug pod in the same namespace
kubectl run debug --image=nicolaka/netshoot -it --rm --namespace=production -- /bin/bash

# From inside the pod, test connectivity
curl -v http://backend.production.svc.cluster.local:8080
nc -zv backend 8080

If you use Cilium, Hubble gives you immediate visibility into policy decisions:

# Watch live flows with policy verdicts
hubble observe --verdict DENIED -f

# See flows for a specific pod
hubble observe --pod backend --type policy

Best Practices for Network Policies

Start with a default-deny in each namespace, then add allow rules incrementally. This forces explicit allowlisting.
Use namespaces for segmentation: Group related workloads in namespaces and use namespace selectors in policies for broad isolation boundaries.
Always allow DNS egress: Forgetting DNS is the most common cause of mysterious application failures after applying egress policies.
Version your policies in Git: Treat Network Policies as code. Review them in pull requests and validate them with tools like kubeconform or kyverno.
Test in staging first: Apply policies in a non-production environment and verify application behavior before promoting to production.
Document policy intent: Add annotations or comments explaining why each policy exists. Future engineers will thank you.
Audit regularly: Periodically review all policies in a namespace to catch drift and overly permissive rules.
Monitor policy violations: Use tools like Hubble or Calico flow logs to detect denied traffic that indicates either an attack or a misconfigured policy.

Conclusion

Troubleshooting Kubernetes Network Policies requires a methodical approach: confirm your CNI supports enforcement, verify label selectors on both pods and namespaces, ensure DNS and egress are allowed, and remember that policies are additive rather than ordered. By combining careful YAML review with observability tools like Hubble or Calico flow logs, you can quickly pinpoint why traffic is being blocked and apply the correct fix. Adopting a default-deny posture with explicit allow rules, version-controlling your policies, and testing changes in staging will keep your cluster both secure and functional, giving you the confidence that your network segmentation works exactly as intended.