Tetragon eBPF Security: Complete Implementation Guide

Introduction to Tetragon eBPF Security

Tetragon is an open-source eBPF-based security observability and runtime enforcement platform developed by Cilium. It leverages the Linux kernel's eBPF technology to provide deep visibility into system calls, network activity, and process execution across containerized and traditional workloads. Unlike traditional security tools that rely on log parsing or agent-based polling, Tetragon operates directly in the kernel, enabling real-time detection and enforcement with minimal overhead.

In this tutorial, you'll learn what Tetragon is, why it matters in modern cloud-native security, how to install and configure it, and how to write custom security policies using its powerful policy framework. We'll also cover best practices for production deployments.

What Is Tetragon?

Tetragon is a flexible security tool that combines two key capabilities: observability and enforcement. It attaches eBPF programs to kernel hooks such as tracepoints, kprobes, and LSM (Linux Security Module) attach points, allowing it to monitor and block security-relevant events as they happen.

Core Components

• eBPF Programs: Loaded into the kernel to capture events like process execution, file access, and network connections.
• Tracing Policies: Kubernetes Custom Resources (CRDs) that define what events to monitor and what actions to take.
• Agent: A daemon that runs on each node, manages eBPF programs, and exports events to logs or SIEM systems.
• Event Pipeline: Filters and enriches events with Kubernetes metadata (namespace, pod, labels) before exporting.

Key Capabilities

• Real-time process execution monitoring with full argument capture
• Network connection tracking at the socket layer
• File and directory access monitoring
• Capability and privilege escalation detection
• Runtime enforcement via SIGKILL or event override
• Low overhead — typically less than 1% CPU per node

Why Tetragon Matters

Modern containerized environments present unique security challenges. Containers share a kernel, meaning a single compromised pod can potentially affect the entire node. Traditional security tools often lack visibility into kernel-level events or introduce significant performance overhead. Tetragon addresses these gaps by operating at the kernel level with eBPF, providing both depth and performance.

Advantages Over Traditional Approaches

• Kernel-level visibility: Sees all syscalls and kernel events regardless of how the application is packaged.
• Real-time enforcement: Can kill processes or block actions before they complete, not just log them after the fact.
• Kubernetes-native: Policies are CRDs, making them versionable and GitOps-friendly.
• Low overhead: eBPF programs run efficiently in kernel space without context switches.
• No application changes: Works transparently without instrumenting application code.

Installing Tetragon

Tetragon can be installed on any Kubernetes cluster running Linux kernel 5.8 or later. The easiest method is using Helm.

Prerequisites

• Kubernetes cluster (1.16+)
• Linux kernel 5.8+ (5.15+ recommended for full LSM support)
• Helm 3.x
• Cluster-admin privileges

Installation with Helm

# Add the Tetragon Helm repository
helm repo add tetragon https://helm.isovalent.com
helm repo update

# Install Tetragon in the tetragon namespace
helm install tetragon tetragon/tetragon \
  --namespace tetragon \
  --create-namespace \
  --set tetragon.enableProcessTracking=true \
  --set tetragon.exportFilename=/var/log/tetragon/tetragon.log

# Verify the installation
kubectl get pods -n tetragon

Verifying the Installation

# Check Tetragon agent status
kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra status

# View live events
kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra getevents

If you see events streaming, Tetragon is successfully installed and monitoring your cluster.

Understanding Tracing Policies

Tetragon policies are defined as Kubernetes Custom Resources called TracingPolicy. Each policy specifies which kernel events to hook, what filters to apply, and what actions to take when events match.

Policy Structure

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: example-policy
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "/bin/bash"
          matchActions:
            - action: Post

Key Fields Explained

• call: The kernel function or syscall to hook (e.g., sys_execve, tcp_connect).
• syscall: Set to true if hooking a syscall.
• args: Defines which arguments to capture and their types.
• selectors: Filtering rules that determine when actions fire.
• matchArgs: Conditions based on argument values.
• matchActions: Actions to take (Post, Sigkill, Override, UnfollowFD).

Practical Examples

Example 1: Detecting Shell Spawns in Containers

One of the most common attack patterns is spawning a shell inside a container that shouldn't need one. This policy detects when /bin/bash or /bin/sh is executed inside any pod.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-shell-spawn
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Postfix"
              values:
                - "/bash"
                - "/sh"
                - "/zsh"
          matchActions:
            - action: Post

Apply this policy with:

kubectl apply -f detect-shell-spawn.yaml

To view the events generated by this policy:

kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra getevents --namespace default | jq '.'

Example 2: Blocking Privilege Escalation

This policy detects and kills any process attempting to use setuid(0) to escalate to root privileges, which is a common post-exploitation technique.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-privilege-escalation
spec:
  kprobes:
    - call: "sys_setuid"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "0"
          matchActions:
            - action: Sigkill

The Sigkill action immediately terminates the offending process, preventing the privilege escalation from completing.

Example 3: Monitoring File Access to Sensitive Paths

This policy monitors read and write access to /etc/shadow, which contains password hashes and should never be accessed by applications.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-shadow-file-access
spec:
  kprobes:
    - call: "sys_openat"
      syscall: true
      args:
        - index: 1
          type: "string"
        - index: 2
          type: "int"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Equal"
              values:
                - "/etc/shadow"
          matchActions:
            - action: Post

Example 4: Tracking Outbound Network Connections

Monitoring outbound connections helps detect data exfiltration and command-and-control communication. This policy tracks all TCP connect calls.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: track-tcp-connections
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sockaddr"
      selectors:
        - matchArgs:
            - index: 0
              operator: "DAddr"
              values:
                - "0.0.0.0/0"
          matchActions:
            - action: Post

Example 5: Detecting Kubernetes Service Account Token Access

Attackers often try to read service account tokens to escalate privileges within the cluster. This policy monitors access to the token file.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-sa-token-access
spec:
  kprobes:
    - call: "sys_openat"
      syscall: true
      args:
        - index: 1
          type: "string"
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/var/run/secrets/kubernetes.io/serviceaccount"
          matchActions:
            - action: Post

Event Export and Integration

Tetragon can export events to multiple destinations for integration with SIEM platforms, log aggregation systems, and alerting tools.

Configuring File Export

helm upgrade tetragon tetragon/tetragon \
  --namespace tetragon \
  --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
  --set tetragon.exportFileMaxSizeMB=100 \
  --set tetragon.exportFileMaxBackups=5 \
  --set tetragon.exportFileCompress=true

Configuring Fluentd Integration

For production deployments, you'll want to forward Tetragon events to a centralized logging system. Here's a sample Fluentd ConfigMap that reads Tetragon logs and forwards them to Elasticsearch:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-tetragon-config
  namespace: tetragon
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/tetragon/tetragon.log
      pos_file /var/log/fluentd-tetragon.pos
      tag tetragon.events
      format json
      read_from_head false
    </source>

    <match tetragon.events>
      @type elasticsearch
      host elasticsearch.logging.svc.cluster.local
      port 9200
      index_name tetragon-events
      type_name _doc
      flush_interval 5s
    </match>

Using the Tetra CLI for Event Queries

# Get all events from the last 5 minutes
kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra getevents --since 5m

# Filter events by namespace
kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra getevents --namespace production

# Filter by event type (process, network, etc.)
kubectl -n tetragon exec daemonset/tetragon -c tetragon -- \
  tetra getevents --type process

Advanced Policy Patterns

Namespace-Scoped Policies

You can restrict policies to specific namespaces using the matchNamespaces selector. This is useful for applying stricter rules to production environments.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: production-shell-block
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchNamespaces:
            - namespace: production
          matchArgs:
            - index: 0
              operator: "Postfix"
              values:
                - "/bash"
                - "/sh"
          matchActions:
            - action: Sigkill

Combining Multiple Conditions

Policies can combine multiple match conditions using AND logic. This policy detects when a process running as root attempts to execute a shell:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: root-shell-execution
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
                - "/bin/zsh"
          matchCapabilities:
            - type: "Effective"
              isNamespaceCapability: false
              operator: "In"
              values:
                - "CAP_SYS_ADMIN"
          matchActions:
            - action: Post

Using LSM Hooks for Enforcement

For kernels 5.15+ with BPF LSM enabled, you can use LSM hooks for more reliable enforcement. LSM hooks fire before the action completes, allowing true prevention rather than detection.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: lsm-block-shell
spec:
  lsms:
    - hook: "bprm_check_security"
      args:
        - index: 0
          type: "linux_binprm"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/bin/bash"
                - "/bin/sh"
          matchActions:
            - action: Override
              argError: -1

The Override action with argError: -1 causes the kernel to return an error code, preventing the shell from executing at all.

Best Practices

1. Start with Observability Before Enforcement

Always begin with Post actions to observe events and tune your policies. Once you're confident the policy catches the right events without false positives, switch to Sigkill or Override for enforcement. This prevents accidental disruption of legitimate workloads.

2. Scope Policies Appropriately

Use namespace selectors, binary matchers, and argument filters to narrow policy scope. Overly broad policies generate excessive events and can impact performance. For example, instead of monitoring all file opens, monitor only access to specific sensitive paths.

3. Test Policies in Staging

Always test new policies in a staging environment before applying them to production. Use the tetra tracepolicy command to validate policy syntax:

# Validate a policy before applying
tetra tracepolicy validate my-policy.yaml

# Dry-run mode (logs events without enforcing)
kubectl apply -f my-policy.yaml --dry-run=server

4. Monitor Tetragon's Resource Usage

While Tetragon is designed for low overhead, complex policies with many selectors can increase resource consumption. Monitor the Tetragon agent's CPU and memory usage:

kubectl -n tetragon top pod -l app.kubernetes.io/name=tetragon

5. Version Control Your Policies

Store all TracingPolicy YAML files in a Git repository and use GitOps tools like ArgoCD or Flux to manage deployments. This ensures policies are auditable, reviewable, and recoverable.

6. Regularly Review Generated Events

Set up dashboards in your SIEM to visualize Tetragon events. Regularly review the top event sources to identify potential security issues or policy tuning opportunities. Key metrics to track include:

• Events per namespace and pod
• Most frequently triggered policies
• Processes generating the most security events
• Network connections to unusual destinations

7. Combine with Other Security Tools

Tetragon is most effective when combined with other security layers. Use it alongside:

• Network policies (Cilium): For network-level segmentation
• Admission controllers (Kyverno, OPA): For pre-deployment policy enforcement
• Image scanning (Trivy): For vulnerability detection before deployment
• Falco: For complementary runtime detection rules

Troubleshooting Common Issues

Tetragon Agent Not Starting

Check that your kernel version meets the minimum requirement and that BPF is enabled:

# Check kernel version
uname -r

# Verify BPF filesystem is mounted
mount | grep bpf

# Check Tetragon agent logs
kubectl -n tetragon logs daemonset/tetragon -c tetragon

Policies Not Generating Events

If your policy isn't generating events, verify the following:

# Check if the policy is loaded
kubectl get tracingpolicy

# Verify the policy status
kubectl describe tracingpolicy my-policy

# Check for eBPF program loading errors
kubectl -n tetragon logs daemonset/tetragon -c tetragon | grep -i error

High CPU Usage

If the Tetragon agent is consuming excessive CPU, review your policies for overly broad matchers. You can also adjust the event export rate:

helm upgrade tetragon tetragon/tetragon \
  --namespace tetragon \
  --set tetragon.exportRateLimit=100 \
  --set tetragon.exportFileMaxSizeMB=50

Conclusion

Tetragon represents a significant advancement in cloud-native security by bringing eBPF-powered kernel-level observability and enforcement to Kubernetes environments. Its ability to detect and block security events in real time, combined with its Kubernetes-native policy model, makes it an essential tool for any organization serious about runtime security. By starting with observability, carefully scoping policies, and integrating with your existing security stack, you can build a robust runtime security posture that catches threats traditional tools miss. As eBPF continues to mature and gain new capabilities, Tetragon will only become more powerful, making now the right time to incorporate it into your security strategy.