Docker with Jenkins: A Complete Guide
Combining Docker with Jenkins unlocks a powerful CI/CD workflow where every build runs in an isolated, reproducible environment. This tutorial walks you through the core concepts, practical setup, best practices, and the most common pitfalls developers encounter when integrating these two technologies.
What It Is: Understanding the Integration
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →Docker with Jenkins means using Docker containers as the execution environment for Jenkins pipelines. Instead of running builds directly on the Jenkins controller or static agent machines, each job—or even each stage—can spin up a fresh container with precisely the tools and dependencies needed. This container exists only for the duration of the build and is destroyed afterward, leaving no lingering artifacts or conflicting software versions on the host.
There are three primary integration patterns:
- Docker as the agent runtime: Jenkins dynamically launches a container to serve as the build agent for an entire pipeline or a specific stage.
- Docker inside the pipeline: The pipeline steps themselves invoke
dockercommands to build, tag, and push images—often from within a Docker-capable agent. - Jenkins itself in Docker: The Jenkins controller and its persistent agents run as Docker containers, making the entire CI infrastructure containerized and portable.
Why It Matters
Without containers, Jenkins agents accumulate software over time. Different projects require conflicting versions of Node.js, Python, or Java. Teams waste hours debugging "works on my machine" issues that stem from subtle environment differences. Docker solves this by making the build environment declarative, version-controlled, and ephemeral. Every commit can be tested against the exact same environment, and that environment can be replicated locally by any developer.
The benefits cascade across the entire delivery lifecycle:
- Reproducibility: The Dockerfile lives in the repository, so the build environment evolves alongside the code.
- Isolation: One pipeline's dependencies never collide with another's.
- Efficiency: Containers spin up in seconds, and Docker layer caching dramatically speeds up repeated builds.
- Clean slate guarantee: Every build starts from a known base image, eliminating "drift" that plagues long-lived agent VMs.
- Scalability: Containerized agents can be orchestrated across Kubernetes clusters, allowing Jenkins to scale elastically.
How to Use It: Practical Setup
Let's walk through a complete setup—from installing Docker on the Jenkins agent to writing a Jenkinsfile that builds and publishes a Docker image.
Step 1: Install Docker and Configure Jenkins
Ensure Docker Engine is installed on the machine that will run your containerized builds. If you are using Jenkins' built-in Docker plugin with dynamic agents, Docker must be available on the Jenkins controller or on the swarm/kubernetes cluster. Verify with:
docker version
docker run hello-world
Install the required Jenkins plugins: Docker Pipeline, Docker, and Docker Commons. These provide the docker global variable and the declarative pipeline syntax for Docker agents.
Step 2: The Dockerfile as Build Environment
Place a Dockerfile in the root of your project repository. This file defines the tools your pipeline needs. For a Node.js project:
# Dockerfile.build
FROM node:20-alpine
RUN apk add --no-cache \
git \
openssh-client \
curl \
jq
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
This image can be pre-built and pushed to a registry, or built on the fly by Jenkins.
Step 3: Declarative Pipeline with Docker Agent
The cleanest approach uses a declarative Jenkinsfile that specifies a Docker image for the entire pipeline or per stage:
pipeline {
agent {
docker {
image 'node:20-alpine'
args '-v /tmp/cache:/tmp/cache -p 3000:3000'
}
}
stages {
stage('Install Dependencies') {
steps {
sh 'npm ci'
}
}
stage('Run Tests') {
steps {
sh 'npm test'
}
}
stage('Build Docker Image') {
steps {
script {
def appImage = docker.build("myapp:${env.BUILD_ID}", "-f Dockerfile.prod .")
appImage.push("latest")
appImage.push(env.BUILD_ID)
}
}
}
}
post {
always {
sh 'docker system prune -f --volumes || true'
}
}
}
Notice that the agent block specifies the image that Jenkins uses to run all steps. The docker.build method inside the script block builds a separate production image from a different Dockerfile—this is the application image that gets deployed, not the build environment.
Step 4: Building Images with Docker in the Pipeline
To build Docker images from within a Jenkins pipeline, you need access to a Docker daemon. The two common approaches are:
Approach A: Mounting the Docker Socket (Docker-outside-of-Docker)
Mount the host's Docker socket into the container. This shares the host's Docker daemon with the pipeline container—fast and uses the host's image cache, but comes with security implications.
pipeline {
agent {
docker {
image 'docker:24-dind'
args '-v /var/run/docker.sock:/var/run/docker.sock'
}
}
stages {
stage('Build and Push') {
steps {
sh 'docker build -t myapp:latest .'
sh 'docker push myapp:latest'
}
}
}
}
Approach B: Docker-in-Docker (DinD)
Run a full Docker daemon inside the container itself. This provides stronger isolation but requires privileged mode and careful volume management.
pipeline {
agent {
docker {
image 'docker:24-dind'
args '--privileged --cpus 2 --memory 4g'
}
}
stages {
stage('Wait for Docker') {
steps {
sh '''
while ! docker info > /dev/null 2>&1; do
echo "Waiting for Docker daemon..."
sleep 2
done
'''
}
}
stage('Build Application Image') {
steps {
sh 'docker build -t registry.example.com/myapp:${BUILD_ID} .'
sh 'docker push registry.example.com/myapp:${BUILD_ID}'
}
}
}
}
Step 5: Using Docker Compose in Pipelines
For integration tests that require multiple services (a database, a cache, a message broker), Docker Compose is invaluable. Your pipeline can spin up the full stack, run tests against it, and tear it down:
pipeline {
agent any
stages {
stage('Integration Tests') {
steps {
sh '''
docker compose -f docker-compose.test.yml up -d
# Wait for services to be healthy
for i in $(seq 1 30); do
if curl -s http://localhost:8080/health | grep -q "ok"; then
break
fi
sleep 2
done
# Run tests
npm run test:integration
'''
}
post {
always {
sh 'docker compose -f docker-compose.test.yml down --volumes --remove-orphans'
}
}
}
}
}
Best Practices
1. Treat Docker Images as Immutable Artifacts
Never install dependencies at deploy time. The Docker image produced by your CI pipeline should be a self-contained artifact that is tested, scanned, and promoted through environments without modification. Use distinct tags per build (e.g., ${BUILD_ID} or the git SHA) rather than overwriting latest.
def imageTag = "${env.BUILD_ID}-${env.GIT_COMMIT.substring(0, 7)}"
def appImage = docker.build("registry.example.com/myapp:${imageTag}")
appImage.push()
2. Leverage Docker Layer Caching Aggressively
Structure your Dockerfiles so that infrequently changing layers come first. Copy dependency manifests before source code so that npm ci or pip install results are cached unless dependencies actually change.
# Good: cache dependencies
FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --production
COPY . .
# ... rest of build
# Bad: invalidates cache on every code change
FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci --production
3. Use Multi-Stage Builds to Keep Images Small
Separate build-time tools from runtime artifacts. This reduces image size, speeds deployment, and shrinks the attack surface.
# Stage 1: build
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
# Stage 2: production runtime
FROM node:20-alpine AS runtime
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package.json ./
EXPOSE 3000
CMD ["node", "dist/server.js"]
4. Never Store Secrets in Images
Use build arguments only for non-sensitive configuration. Secrets like API keys, tokens, and credentials must be passed at runtime via environment variables, mounted files, or a secrets management tool. If you must use a secret during build, leverage Docker BuildKit's --secret flag:
# Jenkinsfile snippet using BuildKit secrets
sh '''
export DOCKER_BUILDKIT=1
docker build \
--secret id=npmrc,src=.npmrc \
-t myapp:${BUILD_ID} \
.
'''
The corresponding Dockerfile uses the --mount=type=secret syntax:
# syntax=docker/dockerfile:1
FROM node:20-alpine
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
npm ci --production
5. Clean Up After Every Build
Docker artifacts accumulate quickly. Dangling images, stopped containers, and orphaned volumes can consume gigabytes of disk space and cause "no space left on device" errors. Add a cleanup step in the post block:
post {
always {
sh '''
docker system prune -f --volumes --filter "until=2h" || true
docker image prune -f --filter "dangling=true" || true
'''
cleanWs()
}
}
6. Pin Exact Image Versions
Never use floating tags like node:latest or python:3 in pipeline agent definitions. A new minor version can break your build silently. Pin to a specific digest or at least a major-minor version:
agent {
docker {
image 'node:20.11.1-alpine@sha256:abc123...'
}
}
7. Run as Non-Root User
By default, most Docker containers run as root. This is unnecessary for CI tasks and can cause permission problems with mounted volumes. Switch to a non-root user in your Dockerfile:
FROM node:20-alpine
RUN addgroup -S appgroup && adduser -S jenkins -G appgroup
USER jenkins
WORKDIR /home/jenkins/app
COPY --chown=jenkins:appgroup . .
RUN npm ci
8. Use a Dedicated Docker Registry for CI Artifacts
Don't push CI-built images directly to production registries without scanning. Use a CI-internal registry as a staging area, then promote images after security scanning and approval. Jenkins can tag and push to different registries per environment:
stage('Promote to Production Registry') {
when {
branch 'main'
}
steps {
script {
def ciImage = "ci-registry.example.com/myapp:${BUILD_ID}"
sh "docker pull ${ciImage}"
sh "docker tag ${ciImage} prod-registry.example.com/myapp:latest"
sh "docker push prod-registry.example.com/myapp:latest"
}
}
}
Common Pitfalls and How to Avoid Them
Pitfall 1: "Cannot connect to the Docker daemon"
Symptom: Your pipeline fails immediately with docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Root cause: The Docker socket is not mounted, Docker-in-Docker hasn't finished starting, or the Docker service isn't running on the host.
Fix: If using Docker-outside-of-Docker, verify the socket mount with -v /var/run/docker.sock:/var/run/docker.sock and ensure the Jenkins user has permission to access it (typically group docker with GID 999). If using DinD, add a wait loop as shown in Step 4 above. Also check that the agent template in Jenkins configuration actually includes the socket volume mount.
Pitfall 2: Disk Space Exhaustion
Symptom: Builds fail with "no space left on device" after running successfully for weeks. The Jenkins agent's disk is full of old images and volumes.
Fix: Implement aggressive cleanup. Beyond the docker system prune in the post block, schedule a cron job on the agent host to run docker system prune -a --volumes --force weekly. Monitor disk usage with a dedicated pipeline stage:
stage('Check Disk Space') {
steps {
sh '''
AVAILABLE=$(df /var/lib/docker | tail -1 | awk '{print $4}')
if [ "$AVAILABLE" -lt 5000000 ]; then
echo "Low disk space! Cleaning..."
docker system prune -a -f --volumes
fi
'''
}
}
Pitfall 3: Layer Cache Misses in CI
Symptom: Every build downloads all dependencies from scratch, even when package.json hasn't changed. Builds take 5-10 minutes longer than expected.
Root cause: The CI system clones the repository into a fresh directory each time, and file timestamps differ. Docker's COPY instruction invalidates the cache if any file metadata changes.
Fix: Use BuildKit with --cache-from and a remote cache. Pull the previous image and reference it as a cache source:
sh '''
export DOCKER_BUILDKIT=1
docker pull registry.example.com/myapp:latest || true
docker build \
--cache-from registry.example.com/myapp:latest \
--build-arg BUILDKIT_INLINE_CACHE=1 \
-t registry.example.com/myapp:${BUILD_ID} \
.
'''
The BUILDKIT_INLINE_CACHE=1 argument embeds cache metadata into the image so future builds can use it. Combine this with a dedicated cache volume mounted at /root/.npm or /cache for package manager caches.
Pitfall 4: Permission Denied on Docker Socket
Symptom: The pipeline runs as a non-root user and gets permission denied when accessing /var/run/docker.sock.
Fix: Add the Jenkins agent user to the docker group on the host, or run the container with the same GID as the host's docker group:
agent {
docker {
image 'my-build-image'
args '-v /var/run/docker.sock:/var/run/docker.sock --group-add 999'
}
}
Alternatively, use sudo inside the container or configure the Docker daemon to listen on a TCP port with TLS—but this adds complexity.
Pitfall 5: Orphaned Containers and Resource Leaks
Symptom: Over time, the host accumulates hundreds of stopped containers and unnamed volumes. Memory and CPU utilization creep upward even when no builds are running.
Fix: Always use --rm when running one-off containers in scripts. In Docker Compose steps, use down --volumes --remove-orphans. For pipeline agent containers, configure Jenkins to remove them automatically by setting removeContainers: true in the Docker plugin configuration.
Pitfall 6: Docker-in-Docker Networking Confusion
Symptom: In a DinD setup, containers you start inside the pipeline cannot reach services on the host or vice versa. Network isolation is stricter than expected.
Fix: Understand that DinD runs a nested Docker daemon with its own network namespace. To allow the inner containers to communicate with the outer world, you may need to run the DinD container with --network host (though this weakens isolation) or explicitly create a shared network. For most CI use cases, Docker-outside-of-Docker (socket mounting) avoids this complexity entirely and is the simpler choice.
Pitfall 7: Hardcoded Credentials in Dockerfiles
Symptom: A security scan finds AWS credentials, private keys, or tokens baked into image layers. These are visible in docker history and compromise your registry.
Fix: Use BuildKit secrets (as shown in Best Practice #4), or move all credential-dependent operations to runtime entrypoint scripts rather than build time. Never COPY .env files containing secrets into an image.
Pitfall 8: Mismatched Docker Versions
Symptom: A pipeline that works on one agent fails on another with obscure API version errors or unsupported feature flags.
Fix: Pin the Docker CLI version in your build image and ensure the Docker daemon version on all agents meets a minimum baseline. Document the required version in a team README. Use the docker:24-dind image (or a specific version) rather than docker:latest for consistency.
Conclusion
Docker and Jenkins together form a remarkably capable CI/CD foundation. When implemented thoughtfully—with pinned image versions, aggressive cleanup, layer caching, multi-stage builds, and proper secret management—the combination eliminates environment drift, speeds up feedback loops, and produces consistent, deployable artifacts. The pitfalls are real but predictable: disk space, socket permissions, cache invalidation, and network quirks all have straightforward solutions once you recognize them. Start with the patterns in this guide, iterate based on your team's specific workflows, and you'll have a pipeline that is both powerful and maintainable for years to come.