Docker BuildKit Features: Production Guide

What is Docker BuildKit?

BuildKit is the next-generation image-building engine for Docker, introduced as a replacement for the legacy builder. It was developed as an open-source project under the Moby umbrella and has been integrated into Docker since version 18.09. BuildKit transforms how container images are constructed by introducing a highly parallel, cache-efficient, and extensible architecture that goes far beyond the simple Dockerfile-processing pipeline of the old builder.

At its core, BuildKit parses Dockerfiles into a Directed Acyclic Graph (DAG) of build operations rather than executing instructions sequentially. This graph-based approach allows independent steps to run concurrently, eliminates unnecessary work through intelligent caching, and provides a robust plugin system for extended functionality. The engine supports features like secrets management, SSH agent forwarding, remote cache import/export, and custom output formats that were simply not possible with the traditional Docker build command.

Why BuildKit Matters for Production

In production environments, image builds are not just a developer convenience — they are a critical part of the CI/CD pipeline. Slow builds delay deployments, insecure handling of credentials exposes infrastructure, and bloated images waste storage and bandwidth. BuildKit directly addresses these production concerns through several fundamental improvements:

Key Advantages Over the Legacy Builder

• Parallel execution of independent stages — Multi-stage builds and unrelated RUN instructions can execute simultaneously, dramatically reducing build times on multi-core machines
• Secrets management without layer leakage — Sensitive data like API tokens and private keys can be mounted into the build context without persisting in image layers
• SSH agent forwarding for private repositories — Build containers can securely access private Git repos and package registries using the host's SSH credentials without copying keys
• Distributed cache with registry support — Build cache can be exported to and imported from container registries, enabling shared caching across CI workers
• Fine-grained cache invalidation — Only stages and instructions that actually changed are rebuilt, avoiding cascade rebuilds
• Custom output formats — Build artifacts can be extracted as tar archives, local directories, or pushed directly to registries without creating dangling intermediate images

Enabling BuildKit

BuildKit is available in Docker 18.09 and later. Starting from Docker 23.0, BuildKit is enabled by default. For older versions or to explicitly ensure it is active, use one of the methods below.

Method 1: Environment Variable

Set the environment variable before running any docker build command. This is ideal for CI pipelines and per-session activation.

export DOCKER_BUILDKIT=1
docker build -t myapp:latest .

Method 2: Daemon Configuration

For permanent enabling across all builds on a machine, edit the Docker daemon configuration file at /etc/docker/daemon.json:

{
  "features": {
    "buildkit": true
  }
}

After saving the file, restart the Docker daemon:

sudo systemctl restart docker

Method 3: Docker Compose

When using docker compose build, you can set the environment variable inline or use the COMPOSE_DOCKER_CLI_BUILD variable along with DOCKER_BUILDKIT:

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose build

Verifying BuildKit is Active

Check the build output for the BuildKit signature. With BuildKit enabled, you will see output lines prefixed with step numbers in brackets and a progress summary at the end, rather than the traditional sequential step output.

BuildKit Features in Depth

1. Secret Management — The --secret Flag

One of the most impactful BuildKit features for production is the ability to expose secrets during build time without baking them into the final image. Secrets are mounted as files into the build container at a specific path and are available only during the execution of the instruction that references them. They never appear in any image layer or in the build history.

Consider a common scenario: downloading dependencies from a private package registry that requires an authentication token. With the legacy builder, the token would either be passed as a build argument (leaking it into image metadata) or copied into the image (persisting in a layer). BuildKit solves this elegantly.

Example: Using Secrets for NPM Authentication

First, create a file containing your secret — for example, an npm auth token:

// .npmrc file (do not commit to version control)
//registry.company.com/:_authToken=${NPM_TOKEN}

Now create a Dockerfile that uses the --mount=type=secret syntax:

# syntax = docker/dockerfile:1

FROM node:20-alpine AS build

WORKDIR /app

# Mount the secret .npmrc file during package installation
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --production

# Copy application code
COPY . .

RUN npm run build

FROM node:20-alpine AS production
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules

EXPOSE 3000
CMD ["node", "dist/server.js"]

Build the image by passing the secret file with the --secret flag:

docker build --secret id=npmrc,src=.npmrc -t myapp:production .

The .npmrc file is mounted into the build container only for the duration of the npm ci command. It does not appear in the /root/.npmrc location of any resulting layer. You can verify this by inspecting the image — the secret file will be absent.

Example: Secrets from Environment Variables

You can also pipe secrets directly from environment variables without writing them to disk:

# On the build host
export API_KEY="sk-prod-abc123secret"

# Pipe the secret via heredoc
docker build --secret id=api_key,src=/dev/stdin -t myapp:production . <<< "$API_KEY"

The corresponding Dockerfile mount:

RUN --mount=type=secret,id=api_key,target=/run/secrets/api_key \
    ./configure-service.sh

2. SSH Agent Forwarding — The --ssh Flag

BuildKit allows forwarding the host's SSH agent socket into build containers. This enables secure access to private Git repositories and SSH-protected services during builds without copying SSH private keys into the image or using insecure build arguments. The SSH credentials are mounted over a Unix socket and are never stored in any image layer.

Example: Cloning Private Git Repositories

Ensure your SSH agent is running and has the appropriate keys loaded:

eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa_production

Dockerfile with SSH mount:

# syntax = docker/dockerfile:1

FROM alpine:3.19 AS builder

# Install git and openssh-client
RUN apk add --no-cache git openssh-client

# Clone private repository using forwarded SSH agent
RUN --mount=type=ssh \
    mkdir -p /workspace && \
    git clone git@github.com:private-org/private-repo.git /workspace/repo

WORKDIR /workspace/repo
RUN ./build.sh

FROM alpine:3.19
COPY --from=builder /workspace/repo/output /app/output
CMD ["/app/output/service"]

Build with SSH forwarding enabled:

docker build --ssh default -t myapp:production .

The --ssh default flag forwards the default SSH agent socket. You can specify multiple named sockets if needed:

docker build --ssh default --ssh admin=~/.ssh/other_sock -t myapp:production .

Example: SSH for Private Go Modules

# syntax = docker/dockerfile:1

FROM golang:1.22 AS builder

RUN --mount=type=ssh \
    git config --global url."git@github.com:".insteadOf "https://github.com/" && \
    go mod download

COPY . .
RUN go build -o /app/server ./cmd/server

FROM debian:bookworm-slim
COPY --from=builder /app/server /usr/local/bin/server
CMD ["server"]

3. Advanced Cache Management

BuildKit introduces a revolutionary caching model that goes far beyond simple layer caching. It supports exporting cache manifests to container registries and importing them in subsequent builds, enabling distributed and persistent caching across ephemeral CI workers.

Cache Export and Import with Registry

To push the build cache to a registry and reuse it later:

# Build and export cache to registry
docker build \
  --cache-to=type=registry,ref=registry.company.com/myapp:cache,mode=max \
  -t myapp:latest .

# Subsequent build imports cache from registry
docker build \
  --cache-from=type=registry,ref=registry.company.com/myapp:cache \
  --cache-to=type=registry,ref=registry.company.com/myapp:cache,mode=max \
  -t myapp:latest .

The mode=max option exports cache for all intermediate layers, not just the final image layers. This maximizes cache reuse even when Dockerfiles change significantly.

Local Cache Export and Import

For local development or CI pipelines with persistent volumes, use the local filesystem cache:

# Export cache to local directory
docker build \
  --cache-to=type=local,dest=./.buildkit-cache \
  -t myapp:latest .

# Import cache from local directory
docker build \
  --cache-from=type=local,src=./.buildkit-cache \
  -t myapp:latest .

Inline Cache (Simpler but Less Powerful)

For simpler setups, you can embed cache metadata directly into the image and push it to a registry:

# Build with inline cache metadata
docker build \
  --cache-to=type=inline \
  -t registry.company.com/myapp:latest \
  .

# Push the image (carries cache metadata)
docker push registry.company.com/myapp:latest

# Subsequent build imports cache from the pushed image
docker build \
  --cache-from=type=registry,ref=registry.company.com/myapp:latest \
  -t registry.company.com/myapp:latest \
  .

Note that inline cache only caches the final image layers, not intermediate stages. For maximum cache efficiency, use the registry cache exporter with mode=max.

Cache Mounts for Package Managers

BuildKit supports persistent cache mounts within build stages to speed up package manager operations. This is particularly powerful for apt, yum, npm, pip, and go mod operations that download dependencies repeatedly across builds.

# syntax = docker/dockerfile:1

FROM node:20-alpine AS build

WORKDIR /app

# Persistent cache for npm modules across builds
RUN --mount=type=cache,target=/root/.npm \
    npm ci

COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

For Python with pip:

# syntax = docker/dockerfile:1

FROM python:3.12-slim

WORKDIR /app

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

COPY . .
CMD ["python", "main.py"]

For Go modules:

# syntax = docker/dockerfile:1

FROM golang:1.22 AS builder

WORKDIR /app

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go mod download

COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    go build -o /app/server ./cmd/server

FROM debian:bookworm-slim
COPY --from=builder /app/server /usr/local/bin/server
CMD ["server"]

4. Parallel Build Stages and Dependency Graphs

BuildKit automatically detects independent build stages and RUN instructions and executes them in parallel. In a multi-stage Dockerfile, stages that do not depend on each other can run simultaneously. This parallelism is transparent — you don't need to configure anything, but structuring your Dockerfile with parallelism in mind yields significant speed improvements.

Example: Parallel Multi-Stage Build

# syntax = docker/dockerfile:1

# Stage 1: Build frontend assets (independent)
FROM node:20-alpine AS frontend
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY frontend/ ./
RUN npm run build

# Stage 2: Build backend binary (independent of frontend)
FROM golang:1.22 AS backend
WORKDIR /app/backend
COPY backend/go.* ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY backend/ ./
RUN go build -o /app/server ./cmd/api

# Stage 3: Combine results (depends on both above)
FROM alpine:3.19 AS production
RUN apk add --no-cache ca-certificates
COPY --from=frontend /app/frontend/dist /var/www/static
COPY --from=backend /app/server /usr/local/bin/server
ENV STATIC_DIR=/var/www/static
EXPOSE 8080
CMD ["server"]

In this Dockerfile, the frontend and backend stages have no dependency on each other. BuildKit will execute them concurrently, potentially halving the build time on a machine with sufficient cores. The legacy builder would process them sequentially, wasting time.

5. Custom Output Formats

BuildKit allows specifying how build results are delivered. Instead of always producing a Docker image loaded into the local daemon, you can export artifacts directly to a directory, a tar archive, or push to a registry without cluttering the local image store.

Export Build Artifacts to Local Directory

# Export the entire build output as a directory
docker build --output type=local,dest=./output .

# Export only a specific stage's output
docker build --target=frontend --output type=local,dest=./static-assets .

Export as Tar Archive

docker build --output type=tar,dest=build-output.tar .

Push Directly to Registry Without Local Image

# Build and push in one step — no local image stored
docker build --output type=registry,ref=registry.company.com/myapp:latest .

This is incredibly useful in CI pipelines where you want to push the image immediately without filling up the local Docker storage on ephemeral build agents.

Combining Output with Cache Export

docker build \
  --cache-from=type=registry,ref=registry.company.com/myapp:cache \
  --cache-to=type=registry,ref=registry.company.com/myapp:cache,mode=max \
  --output type=registry,ref=registry.company.com/myapp:latest \
  .

6. Heredoc Syntax for Multi-Line Scripts

BuildKit supports heredoc syntax in Dockerfiles, allowing you to write multi-line scripts cleanly without backslash chains or separate script files. This improves readability and maintainability of complex RUN instructions.

# syntax = docker/dockerfile:1

FROM ubuntu:22.04

RUN <

# syntax = docker/dockerfile:1

FROM debian:bookworm-slim AS production
# ... rest of Dockerfile

docker build \
  --cache-from=type=registry,ref=${CACHE_IMAGE}:cache \
  --cache-to=type=registry,ref=${CACHE_IMAGE}:cache,mode=max \
  --secret id=npm_token,src=/run/secrets/npm_token \
  --ssh default \
  --output type=registry,ref=${PRODUCTION_IMAGE}:${TAG} \
  --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --build-arg VCS_REF=$(git rev-parse --short HEAD) \
  .

# syntax = docker/dockerfile:1

FROM node:20-alpine@sha256:abc123...def456 AS build
# Explicit digest ensures reproducible builds

WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --production --ignore-scripts
# ... rest of build

# Capture build logs with timestamps for audit trail
docker build --progress=plain --debug . 2>&1 | tee build-log-$(date +%s).txt

RUN --mount=type=ssh \
    --mount=type=secret,id=npm_token,target=/root/.npmrc \
    --mount=type=secret,id=pypi_token,target=/root/.pypirc \
    --mount=type=cache,target=/root/.npm \
    npm ci && \
    pip install -r requirements.txt

# syntax = docker/dockerfile:1

# === Stage 1: Frontend Build ===
FROM node:20-alpine@sha256:abc123 AS frontend-builder

WORKDIR /app/frontend

COPY frontend/package.json frontend/package-lock.json ./

RUN --mount=type=cache,target=/root/.npm \
    npm ci --production=false

COPY frontend/ ./

RUN npm run build

# === Stage 2: Backend Build ===
FROM golang:1.22@sha256:def456 AS backend-builder

WORKDIR /app/backend

COPY backend/go.mod backend/go.sum ./

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go mod download

COPY backend/ ./

RUN --mount=type=cache,target=/go/pkg/mod \
    go build -ldflags="-w -s" -o /app/server ./cmd/api

# === Stage 3: Production Image Assembly ===
FROM alpine:3.19@sha256:ghi789 AS production

RUN <

docker build \
  --cache-from=type=registry,ref=registry.company.com/myapp:build-cache \
  --cache-to=type=registry,ref=registry.company.com/myapp:build-cache,mode=max \
  --secret id=npm_token,src=/run/ci-secrets/npm_token \
  --ssh default \
  --output type=registry,ref=registry.company.com/myapp:${CI_COMMIT_SHA} \
  --build-arg BUILD_DATE=${BUILD_DATE} \
  --build-arg VCS_REF=${CI_COMMIT_SHA} \
  --progress=plain \
  .