Varnish Cache: Complete Setup and Configuration Guide

Introduction to Varnish Cache

Varnish Cache is a high-performance HTTP accelerator and reverse proxy designed to dramatically speed up dynamic, content-heavy web applications. Written in C and built around a sophisticated event-driven architecture, Varnish sits between your users and your backend web servers, caching responses in memory and serving subsequent requests directly without touching the origin server.

Unlike many caching solutions that operate at the application layer, Varnish operates purely at the HTTP level. It does not understand your database, your templating engine, or your business logic. Instead, it treats every request as an HTTP transaction and makes intelligent decisions about what to cache, how long to store it, and when to invalidate it. This simplicity is what makes Varnish extraordinarily fast — it can serve hundreds of thousands of requests per second from a single modest server.

Why Varnish Matters

Modern web applications face increasing pressure to deliver content quickly. Users expect sub-second load times, search engines penalize slow sites, and traffic spikes during promotions or viral events can overwhelm even well-provisioned backends. Varnish addresses these challenges by:

Reducing backend load: Cached responses never reach your application servers, freeing them to handle uncached work.
Lowering latency: Memory-based serving eliminates database queries, template rendering, and network hops to the origin.
Handling traffic spikes: A single Varnish instance can absorb massive request volumes that would otherwise crash your stack.
Providing flexible routing: Varnish can route requests to different backends based on URL patterns, headers, or cookies.
Enabling graceful degradation: When backends fail, Varnish can serve stale content instead of returning errors.

How Varnish Works

At its core, Varnish is a reverse proxy that intercepts incoming HTTP requests, checks its cache for a matching response, and either serves the cached object immediately or forwards the request to a backend server. The decision-making logic is controlled by a domain-specific language called VCL (Varnish Configuration Language).

The request lifecycle flows through several stages. When a request arrives, Varnish parses it and executes the vcl_recv subroutine, where you decide whether to attempt a cache lookup, pass the request to the backend, or pipe it directly. If a cache lookup is attempted, Varnish checks its in-memory store. On a cache hit, vcl_hit runs and the cached object is delivered through vcl_deliver. On a miss, vcl_miss runs, the request goes to the backend via vcl_backend_fetch, and the response is processed through vcl_backend_response before being stored and delivered.

Varnish stores cached objects in a shared memory segment called a slab. This design avoids the overhead of copying data between user space and kernel space, which is one reason Varnish achieves such high throughput. The cache size is configured at startup and typically consumes a large portion of available RAM.

Installing Varnish

Varnish is available in the package repositories of most Linux distributions. The examples below assume a Debian or Ubuntu-based system, but equivalent commands exist for RHEL, CentOS, and other distributions.

Installation on Debian/Ubuntu

# Add the official Varnish repository
curl -s https://packagecloud.io/install/repositories/varnishcache/varnish71/script.deb.sh | sudo bash

# Install Varnish
sudo apt-get update
sudo apt-get install varnish

# Verify installation
varnishd -V

Installation on RHEL/CentOS

# Add the official Varnish repository
curl -s https://packagecloud.io/install/repositories/varnishcache/varnish71/script.rpm.sh | sudo bash

# Install Varnish
sudo yum install varnish

# Verify installation
varnishd -V

After installation, the main configuration files are typically located at /etc/varnish/default.vcl for the VCL configuration and /etc/default/varnish or /etc/varnish/varnish.params for daemon parameters such as storage size and listen port.

Basic Configuration

The default VCL file provides a sensible starting point, but real-world deployments require customization. Let us build a configuration step by step. First, examine the default daemon parameters to understand how Varnish is launched.

Daemon Parameters

On systemd-based systems, Varnish parameters are often defined in /etc/varnish/varnish.params:

# /etc/varnish/varnish.params
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_STORAGE="malloc,2G"
VARNISH_TTL=120

The VARNISH_STORAGE parameter defines how Varnish stores cached objects. The malloc storage type uses system memory and is the most common choice for production. The value 2G allocates 2 gigabytes. Alternatively, you can use file storage for disk-backed caching, though this is slower and generally discouraged for performance-critical workloads.

A Minimal VCL Configuration

Here is a minimal but functional VCL file that caches static assets and passes dynamic requests to a backend:

# /etc/varnish/default.vcl

vcl 4.1;

import std;

# Define the backend server
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Bypass cache for POST, PUT, DELETE requests
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Strip cookies for static assets so they can be cached
    if (req.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$") {
        unset req.http.Cookie;
        return (hash);
    }

    # Pass requests with authorization headers
    if (req.http.Authorization) {
        return (pass);
    }

    return (hash);
}

sub vcl_backend_response {
    # Cache static assets for 1 hour regardless of backend headers
    if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$") {
        unset beresp.http.Set-Cookie;
        set beresp.ttl = 1h;
    }

    # Do not cache responses with Set-Cookie headers
    if (beresp.http.Set-Cookie) {
        set beresp.uncacheable = true;
        set beresp.ttl = 120s;
    }
}

sub vcl_deliver {
    # Add a header to indicate cache status for debugging
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

This configuration defines a single backend listening on port 8080, which is where your web server or application server should run. Varnish itself listens on port 80 and receives all incoming traffic. Static assets have their cookies stripped so they become cacheable, and a debug header is added to responses so you can verify whether content was served from cache.

Advanced VCL Techniques

Multiple Backends and Load Balancing

Varnish can distribute requests across multiple backend servers using directors. The vmod_directors module provides round-robin, random, and hash-based directors.

vcl 4.1;

import std;
import directors;

backend server1 {
    .host = "10.0.0.11";
    .port = "8080";
    .max_connections = 200;
    .connect_timeout = 3s;
    .first_byte_timeout = 20s;
    .between_bytes_timeout = 5s;
}

backend server2 {
    .host = "10.0.0.12";
    .port = "8080";
    .max_connections = 200;
    .connect_timeout = 3s;
    .first_byte_timeout = 20s;
    .between_bytes_timeout = 5s;
}

sub vcl_init {
    new cluster = directors.round_robin();
    cluster.add_backend(server1);
    cluster.add_backend(server2);
}

sub vcl_recv {
    set req.backend_hint = cluster.backend();
    return (hash);
}

Health Checks and Grace Mode

To ensure Varnish only sends traffic to healthy backends and can serve stale content during outages, configure health probes and grace periods:

backend server1 {
    .host = "10.0.0.11";
    .port = "8080";
    .probe = {
        .url = "/health";
        .timeout = 2s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

sub vcl_backend_response {
    # Keep objects in cache for up to 24 hours after they expire
    set beresp.grace = 24h;
}

sub vcl_hit {
    # Serve stale content if the backend is sick
    if (obj.ttl >= 0s) {
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 10s > 0s) {
            return (deliver);
        }
    } else {
        if (obj.ttl + obj.grace > 0s) {
            return (deliver);
        }
    }
    return (miss);
}

The probe checks the /health endpoint every 5 seconds. A backend is considered healthy if at least 3 of the last 5 checks succeed. Grace mode allows Varnish to serve expired content when the backend is unavailable, providing a seamless experience during outages.

Cache Invalidation with Bans

Varnish supports bans, which are rules that mark cached objects as stale without immediately purging them. Bans are useful for invalidating groups of objects matching a pattern:

# Purge a single URL
curl -X PURGE http://localhost:6081/about.html

# Ban all objects from a specific host
curl -X BAN http://localhost:6081/ -H "X-Ban-Host: example.com"

To handle these requests in VCL, add the following logic:

acl purge_acl {
    "localhost";
    "127.0.0.1";
    "10.0.0.0"/24;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (!client.ip ~ purge_acl) {
            return (synth(403, "Forbidden"));
        }
        return (purge);
    }

    if (req.method == "BAN") {
        if (!client.ip ~ purge_acl) {
            return (synth(403, "Forbidden"));
        }
        if (!req.http.X-Ban-Host) {
            return (synth(400, "Missing X-Ban-Host header"));
        }
        ban("obj.http.host == " + req.http.X-Ban-Host);
        return (synth(200, "Ban added"));
    }

    return (hash);
}

Handling Cookies and Personalized Content

One of the most common challenges with Varnish is caching pages that include personalized elements. A common strategy is to strip non-essential cookies and only bypass the cache when specific session cookies are present:

sub vcl_recv {
    # Remove Google Analytics cookies
    if (req.http.Cookie) {
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm[a-z]=[0-9.]+;?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[0-9.]+;?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "_gid=[0-9.]+;?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^\s*;", "");
        set req.http.Cookie = regsuball(req.http.Cookie, ";\s*;", ";");
    }

    # If no cookies remain, unset the header entirely
    if (req.http.Cookie ~ "^\s*$") {
        unset req.http.Cookie;
    }

    # Bypass cache for authenticated users
    if (req.http.Cookie ~ "session_id=") {
        return (pass);
    }

    return (hash);
}

Monitoring and Troubleshooting

Varnish provides several tools for monitoring cache performance and diagnosing issues. The most important is varnishstat, which displays real-time statistics about cache hits, misses, backend connections, and memory usage.

# View live statistics
varnishstat

# View statistics with a 1-second refresh
varnishstat -1

# Log requests in real time
varnishlog

# Log only backend requests
varnishlog -b

# Log only client requests
varnishlog -c

# View request history
varnishncsa

Key metrics to watch in varnishstat include cache_hit and cache_miss, which indicate your hit ratio. A healthy deployment typically achieves a hit ratio above 80% for static content and 50% or higher for dynamic content. The n_lru_nuked counter indicates how many objects were evicted to make room for new ones; a high value suggests you should increase the cache size.

For deeper analysis, varnishlog provides a detailed transaction log showing every step Varnish takes for each request. This is invaluable for debugging why a particular response is not being cached. Look for the TTL and VCL_return tags to understand caching decisions.

Best Practices

Allocate sufficient memory: Size your cache to hold your working set of objects. Monitor n_lru_nuked and increase storage if objects are being evicted prematurely.
Set appropriate TTLs: Let your backend send proper Cache-Control and Expires headers, and use VCL to override only when necessary. This keeps caching logic close to the content source.
Use grace mode: Always configure a grace period so Varnish can serve stale content during backend failures or slow responses.
Restrict purge access: Only allow trusted IPs to issue purge and ban requests. Exposing these endpoints publicly can be abused to degrade cache effectiveness.
Strip unnecessary cookies: Analytics, A/B testing, and other third-party cookies often prevent caching. Remove them in vcl_recv for cacheable content.
Use ESI for partial caching: Edge Side Includes allow you to cache page fragments independently, enabling personalized pages to be assembled from cached components.
Monitor hit ratios: Regularly review varnishstat output and set up alerts for sudden drops in cache hit rates, which may indicate configuration problems or backend issues.
Test VCL before deploying: Use varnishd -C -f default.vcl to compile and validate your VCL without restarting the daemon.
Keep VCL modular: For complex configurations, use include statements to split VCL into manageable files organized by function.
Secure the admin port: The management interface on port 6082 should never be exposed to the public internet. Bind it to localhost or a private network only.

Validating and Reloading Configuration

# Validate VCL syntax without applying
varnishd -C -f /etc/varnish/default.vcl > /dev/null

# Reload VCL without dropping cache (recommended)
varnishadm vcl.load new_config /etc/varnish/default.vcl
varnishadm vcl.use new_config

# Or use the reload utility
varnishreload

The vcl.load and vcl.use approach is preferred in production because it compiles the new configuration and atomically switches to it without restarting the daemon or losing cached objects.

Conclusion

Varnish Cache is a powerful tool that can transform the performance and scalability of web applications. By sitting in front of your backend servers and serving cached responses from memory, it reduces latency, lowers infrastructure costs, and provides resilience during traffic spikes and backend failures. While the VCL language has a learning curve, the investment pays off quickly through fine-grained control over caching behavior. Start with a simple configuration, monitor your hit ratios, and iteratively refine your VCL to handle cookies, personalization, and cache invalidation. With proper tuning and adherence to best practices, Varnish can easily deliver order-of-magnitude performance improvements for most web workloads.