Docker with Jenkins: Production Guide

Docker with Jenkins: A Production Deployment Guide

Combining Docker with Jenkins forms one of the most powerful CI/CD pipelines available to modern development teams. This guide walks you through every aspect of setting up, configuring, and hardening a production-ready Jenkins environment powered by Docker—from initial containerization to advanced multi-stage pipelines, security hardening, and disaster recovery.

What This Integration Actually Means

At its core, running Jenkins with Docker means the Jenkins master itself runs inside a container, and every build job spins up ephemeral agent containers that are destroyed after execution. This is fundamentally different from installing Jenkins on a bare-metal server or virtual machine. The Docker approach treats your entire CI infrastructure as code—versioned, reproducible, and portable.

The architecture typically follows one of two patterns:

Docker-in-Docker (DinD): The Jenkins container runs a Docker daemon inside itself, allowing it to build and run nested containers. This is simple but carries security trade-offs.
Docker-out-of-Docker (DooD): The Jenkins container mounts the host's Docker socket (/var/run/docker.sock) and uses the host's Docker daemon directly. This is faster but requires careful permission management.
Remote Docker Engine: Jenkins connects to a remote Docker daemon over TCP, often with TLS mutual authentication. This is the most secure production pattern.

Why Docker + Jenkins Matters in Production

Running Jenkins on a traditional server introduces dependency drift. Over months, the server accumulates packages, manual fixes, and configuration tweaks that are undocumented. A disaster recovery scenario becomes a guessing game. Docker eliminates this problem entirely—your Jenkins setup is defined in a Dockerfile and a compose file, rebuildable in minutes.

The real production benefits include:

Build environment parity: Your Jenkins agents can use the exact same Docker images that developers use locally. No more "works on my machine" discrepancies between dev laptops and CI.
Resource isolation: Each build job runs in its own container with defined CPU and memory limits. A rogue build cannot consume all server resources and starve other jobs.
Artifact immutability: Docker images built in CI are tagged with commit SHAs and pushed to a registry. They become immutable deployment artifacts that flow directly to staging and production.
Secret management: Build credentials never touch agent disk storage—they are injected as environment variables into ephemeral containers that vanish after the build completes.
Horizontal scaling: When build queues grow, you can spin up additional Jenkins agent containers on any Docker-capable node in your cluster without installing Java or Jenkins-specific tooling.

Setting Up Jenkins in Docker for Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

1. Writing a Production Dockerfile for Jenkins Master

The official Jenkins image from Docker Hub is a solid starting point, but production environments require custom hardening. You need to pin versions, lock down plugins, pre-configure security settings, and ensure the container runs as a non-root user.

Below is a production-grade Dockerfile that builds a customized Jenkins master image:

# Dockerfile for production Jenkins master
FROM jenkins/jenkins:2.492.1-lts-jdk17

USER root

# Install required system packages
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 \
    python3-pip \
    curl \
    wget \
    unzip \
    jq \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# Install Docker CLI only (no daemon) for remote Docker engine communication
RUN curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz \
    | tar xzv --strip-components=1 -C /usr/local/bin docker/docker \
    && chmod +x /usr/local/bin/docker

# Add HashiCorp Vault CLI for secret retrieval
RUN curl -fsSL https://releases.hashicorp.com/vault/1.18.1/vault_1.18.1_linux_amd64.zip \
    -o vault.zip && unzip vault.zip -d /usr/local/bin/ && rm vault.zip

# Create Jenkins config directory structure
RUN mkdir -p /var/jenkins_home/init.groovy.d \
    && mkdir -p /var/jenkins_home/casc_configs \
    && mkdir -p /var/jenkins_backup

# Copy Configuration-as-Code and init scripts
COPY ./jenkins-casc.yaml /var/jenkins_home/casc_configs/jenkins.yaml
COPY ./init.groovy.d/ /var/jenkins_home/init.groovy.d/
COPY ./plugin-list.txt /tmp/plugin-list.txt
COPY ./entrypoint.sh /usr/local/bin/jenkins-entrypoint.sh

# Set ownership for Jenkins user
RUN chown -R jenkins:jenkins /var/jenkins_home \
    && chown -R jenkins:jenkins /var/jenkins_backup \
    && chmod +x /usr/local/bin/jenkins-entrypoint.sh

USER jenkins

# Pre-install plugins from pinned versions file
RUN jenkins-plugin-cli --plugin-file /tmp/plugin-list.txt --verbose

# Configure JVM options for production workloads
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false \
  -Djava.awt.headless=true \
  -Xmx4g \
  -Xms2g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:+PrintGCDetails \
  -XX:+PrintGCTimeStamps \
  -Xlog:gc:/var/jenkins_home/gc.log \
  -Dhudson.security.SecurityFix.REALM=production \
  -Dcom.cloudbees.jenkins.GitHubPushFilter.allowUnescapedCharacters=false"

# Expose ports
EXPOSE 8080 50000

ENTRYPOINT ["/usr/local/bin/jenkins-entrypoint.sh"]

The key production decisions in this Dockerfile are:

Pinned LTS version: Never use :latest in production. Pin to a specific LTS release like 2.492.1-lts-jdk17 so that image updates are deliberate and tested.
Minimal surface area: The --no-install-recommends flag and cleanup commands reduce the image attack surface. The final image should contain only what Jenkins needs to operate.
Non-root execution: The USER jenkins directive ensures the main process does not run as root, limiting the blast radius of any potential exploit.
GC logging: The JVM flags include garbage collection logging. In production, you need visibility into JVM performance to diagnose memory pressure before it causes outages.
Plugin version pinning: The plugin-list.txt file should contain exact plugin versions, not just names. This prevents a rebuild from pulling incompatible plugin updates.

2. The Plugin Version File

Create a plugin-list.txt with pinned versions. This is critical—without it, a Docker rebuild can pull newer plugin versions that introduce breaking changes or vulnerabilities in the opposite direction:

# plugin-list.txt - Production pinned plugins for Jenkins 2.492.1
blueocean:1.27.16
configuration-as-code:1775.v810dcb6db5f9
docker-workflow:599.v9c_537f7e5b_25
git:5.2.1
github-branch-source:1767.v7a_22a_b_49a_cd6
job-dsl:1.87
kubernetes:4306.vc91e951ea_358
pipeline-utility-steps:2.17.0
workflow-aggregator:596.v8c6e9b_3a_6b_5e
timestamper:1.26
ldap:711.vb_d00715ee0e1
matrix-auth:3.2.2
mailer:472.vf7c2891b_3302
dark-theme:365.vb_3b_7b_ea_5159
role-strategy:727.v6b_0172a_60b_41

3. Configuration-as-Code YAML

The Configuration-as-Code plugin (CasC) lets you define Jenkins' entire system configuration in a YAML file. This is the single most important production pattern—it makes Jenkins configuration auditable, version-controlled, and reproducible:

# jenkins-casc.yaml - Production Jenkins system configuration
jenkins:
  systemMessage: "Production Jenkins CI/CD Server - Managed via Configuration-as-Code"
  numExecutors: 0
  scmCheckoutRetryCount: 3
  quietPeriod: 5
  slaveAgentPort: 50000
  slaveAgentPortEnforce: true

  securityRealm:
    ldap:
      server: "ldap.internal.company.com"
      rootDN: "dc=company,dc=com"
      userSearchBase: "ou=people"
      userSearch: "uid={0}"
      groupSearchBase: "ou=groups"
      managerDN: "cn=jenkins-bind,ou=service-accounts,dc=company,dc=com"
      managerPasswordSecret: "${LDAP_BIND_PASSWORD}"
      displayNameAttribute: "cn"
      mailAddressAttribute: "mail"

  authorizationStrategy:
    roleBased:
      roles:
        global:
          - name: "admin"
            pattern: "admin-team"
            permissions:
              - "Overall/Administer"
          - name: "developer"
            pattern: "engineering-team"
            permissions:
              - "Overall/Read"
              - "Job/Read"
              - "Job/Build"
              - "Job/Workspace"
              - "View/Read"

  globalLibraries:
    libraries:
      - name: "production-shared-library"
        defaultVersion: "v2.3.1"
        implicit: true
        retriever:
          modernSCM:
            github:
              repoOwner: "myorg"
              repository: "jenkins-shared-lib"
              credentialsId: "github-app-token"

  views:
    - view:
        name: "Production-Pipelines"
        type: "ListView"
        filterExecutors: true
        filterQueue: true
        columns:
          - column:
              type: "StatusColumn"
          - column:
              type: "WeatherColumn"
          - column:
              type: "JobColumn"

tool:
  jdk:
    installations:
      - name: "jdk-17"
        properties:
          - installSource:
              installers:
                - extractZip:
                    url: "https://download.oracle.com/java/GA/jdk17/..."
                    subdir: "jdk-17.0.12"
      - name: "jdk-21"
        properties:
          - installSource:
              installers:
                - extractZip:
                    url: "https://download.oracle.com/java/GA/jdk21/..."
                    subdir: "jdk-21.0.5"

  maven:
    installations:
      - name: "maven-3.9"
        properties:
          - installSource:
              installers:
                - maven:
                    mavenVersion: "3.9.9"

  docker:
    installations:
      - name: "docker-remote"
        properties:
          - installSource:
              installers:
                - dockerFromBinary:
                    label: "linux-amd64"
                    binaryUrl: "https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz"

credentials:
  system:
    credentials:
      - string:
          scope: GLOBAL
          id: "docker-registry-password"
          secret: "${DOCKER_REGISTRY_PASSWORD}"
          description: "Production Docker registry credential"
      - gitHubApp:
          scope: GLOBAL
          id: "github-app-token"
          appId: "${GITHUB_APP_ID}"
          privateKey: "${GITHUB_APP_PRIVATE_KEY}"
          owner: "myorg"
          description: "GitHub App for production repositories"

security:
  globalJobDslSecurityConfiguration:
    useGroovySandbox: true
  remotingCLI:
    enabled: false
  scriptApproval:
    approvedSignatures:
      - "method java.util.Map get java.lang.Object"
      - "new java.util.HashMap"

jobs:
  - script: >
      folder('production-pipelines') {
          displayName('Production CI/CD Pipelines')
          description('Contains all production-grade build and deployment jobs')
      }

The CasC file references environment variables like ${LDAP_BIND_PASSWORD}. These are injected at container startup, never hardcoded. This separation of configuration and secrets is essential for production security.

4. The Custom Entrypoint Script

A custom entrypoint script handles pre-flight checks, secret retrieval, and graceful shutdown. This is where you integrate with secret stores like HashiCorp Vault or AWS Secrets Manager:

#!/bin/bash
# entrypoint.sh - Production Jenkins entrypoint script

set -euo pipefail

log() {
  echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] [JENKINS-ENTRYPOINT] $1"
}

log "Starting production Jenkins entrypoint sequence"

# Verify required environment variables
REQUIRED_VARS=("JENKINS_HOME" "DOCKER_REGISTRY_PASSWORD" "LDAP_BIND_PASSWORD" "GITHUB_APP_ID" "GITHUB_APP_PRIVATE_KEY")
MISSING_VARS=()

for var in "${REQUIRED_VARS[@]}"; do
  if [[ -z "${!var:-}" ]]; then
    MISSING_VARS+=("$var")
  fi
done

if [[ ${#MISSING_VARS[@]} -gt 0 ]]; then
  log "ERROR: Missing required environment variables: ${MISSING_VARS[*]}"
  exit 1
fi

# Attempt secret retrieval from Vault if configured
if [[ -n "${VAULT_ADDR:-}" ]]; then
  log "Retrieving secrets from HashiCorp Vault at ${VAULT_ADDR}"
  
  export VAULT_TOKEN=$(curl -s --request POST \
    --data "{\"role\": \"jenkins-master\"}" \
    "${VAULT_ADDR}/v1/auth/kubernetes/login" | jq -r '.auth.client_token')
  
  if [[ -z "${VAULT_TOKEN}" || "${VAULT_TOKEN}" == "null" ]]; then
    log "ERROR: Failed to authenticate with Vault"
    exit 1
  fi
  
  DOCKER_REGISTRY_PASSWORD=$(curl -s --header "X-Vault-Token: ${VAULT_TOKEN}" \
    "${VAULT_ADDR}/v1/secret/data/ci/docker-registry" | jq -r '.data.data.password')
  
  LDAP_BIND_PASSWORD=$(curl -s --header "X-Vault-Token: ${VAULT_TOKEN}" \
    "${VAULT_ADDR}/v1/secret/data/ci/ldap" | jq -r '.data.data.bind_password')
  
  log "Secrets retrieved successfully from Vault"
fi

# Perform filesystem health check on JENKINS_HOME
if [[ ! -w "${JENKINS_HOME}" ]]; then
  log "ERROR: JENKINS_HOME directory is not writable"
  exit 1
fi

DISK_SPACE=$(df -m "${JENKINS_HOME}" | awk 'NR==2 {print $4}')
if [[ ${DISK_SPACE} -lt 2048 ]]; then
  log "WARNING: Low disk space on JENKINS_HOME: ${DISK_SPACE}MB available. Minimum recommended: 2048MB"
fi

# Check connectivity to Docker registry
log "Testing Docker registry connectivity"
if ! curl -s --max-time 10 "https://registry.internal.company.com/v2/" > /dev/null 2>&1; then
  log "WARNING: Cannot reach Docker registry. Builds may fail."
fi

# Set up graceful shutdown handler
cleanup() {
  log "Received termination signal. Shutting down Jenkins gracefully..."
  
  # Trigger Jenkins safe shutdown via API
  if curl -s -X POST "http://localhost:8080/safeExit" --max-time 30; then
    log "Jenkins safe shutdown initiated"
  else
    log "WARNING: Could not trigger safe shutdown API, Jenkins may exit forcefully"
  fi
  
  # Wait for Jenkins to exit cleanly
  sleep 30
  log "Cleanup complete"
}

trap cleanup SIGTERM SIGINT SIGHUP

# Launch Jenkins with the CasC configuration
log "Launching Jenkins master with Configuration-as-Code"
exec java ${JAVA_OPTS} \
  -Djenkins.home="${JENKINS_HOME}" \
  -Dcasc.jenkins.config="/var/jenkins_home/casc_configs/jenkins.yaml" \
  -Djava.io.tmpdir="/tmp" \
  -jar /usr/share/jenkins/jenkins.war \
  --httpPort=8080 \
  --prefix="${JENKINS_PREFIX:-/}" \
  "$@"

5. Docker Compose for Production Orchestration

A production Docker Compose file ties everything together with resource limits, restart policies, volume management, and networking. This file should live in version control alongside the Dockerfile:

# docker-compose.yml - Production Jenkins stack
version: "3.8"

services:
  jenkins-master:
    image: registry.internal.company.com/jenkins-master:2.492.1-prod-20250115
    container_name: jenkins-production-master
    restart: unless-stopped
    hostname: jenkins.internal.company.com
    user: "1000:1000"
    
    ports:
      - "8080:8080"
      - "50000:50000"
    
    volumes:
      - jenkins_home:/var/jenkins_home
      - jenkins_backup:/var/jenkins_backup
      - /var/run/docker.sock:/var/run/docker.sock:ro
    
    environment:
      - TZ=UTC
      - JENKINS_HOME=/var/jenkins_home
      - JENKINS_PREFIX=/
      - VAULT_ADDR=https://vault.internal.company.com
      - DOCKER_REGISTRY_PASSWORD=${DOCKER_REGISTRY_PASSWORD}
      - LDAP_BIND_PASSWORD=${LDAP_BIND_PASSWORD}
      - GITHUB_APP_ID=${GITHUB_APP_ID}
      - GITHUB_APP_PRIVATE_KEY=${GITHUB_APP_PRIVATE_KEY}
      - CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs/jenkins.yaml
      - DOCKER_HOST=tcp://docker-engine.internal.company.com:2376
      - DOCKER_TLS_VERIFY=1
      - DOCKER_CERT_PATH=/var/jenkins_home/docker-certs
    
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: "8192m"
        reservations:
          cpus: "2"
          memory: "4096m"
    
    healthcheck:
      test: ["CMD", "curl", "-f", "-s", "--max-time", "10", "http://localhost:8080/login"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s
    
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"
        compress: "true"
    
    networks:
      - jenkins-network
      - registry-network

  jenkins-agent-java:
    image: registry.internal.company.com/jenkins-agent-java:17-20250115
    restart: unless-stopped
    user: "1000:1000"
    
    environment:
      - JENKINS_URL=http://jenkins-master:8080
      - JENKINS_SECRET=${AGENT_JAVA_SECRET}
      - JENKINS_AGENT_NAME=agent-java-${HOSTNAME}
      - JENKINS_AGENT_WORKDIR=/home/jenkins/agent
      - TZ=UTC
    
    volumes:
      - agent_java_workspace:/home/jenkins/agent/workspace
      - agent_java_cache:/home/jenkins/.m2
      - /var/run/docker.sock:/var/run/docker.sock:ro
    
    deploy:
      resources:
        limits:
          cpus: "3"
          memory: "6144m"
        reservations:
          cpus: "1"
          memory: "2048m"
    
    networks:
      - jenkins-network

volumes:
  jenkins_home:
    driver: local
    driver_opts:
      type: nfs4
      device: ":/export/jenkins/production/home"
      o: "addr=nfs-storage.internal.company.com,rw,noatime,hard,timeo=600,retrans=3"
  jenkins_backup:
    driver: local
    driver_opts:
      type: nfs4
      device: ":/export/jenkins/production/backup"
      o: "addr=nfs-storage.internal.company.com,rw,noatime,hard,timeo=600,retrans=3"
  agent_java_workspace:
    driver: local
  agent_java_cache:
    driver: local

networks:
  jenkins-network:
    driver: bridge
    internal: true
    ipam:
      config:
        - subnet: 172.30.0.0/16
  registry-network:
    driver: bridge
    internal: false

Notice the production details: NFS-backed volumes for persistence, resource limits with reservations, health checks that validate Jenkins is actually responsive, log rotation, and separate internal networks that restrict which services can communicate. The Docker socket is mounted read-only (:ro) as a defense-in-depth measure.

Building a Production Pipeline with Docker Agents

Declarative Pipeline with Ephemeral Docker Agents

The true power of Docker + Jenkins emerges in declarative pipelines where each stage runs in its own isolated container. This ensures no tooling contamination between stages and makes the pipeline self-documenting:

// Jenkinsfile - Production multi-stage Docker pipeline
pipeline {
    agent none
    
    environment {
        DOCKER_REGISTRY = 'registry.internal.company.com'
        DOCKER_ORG = 'production'
        APP_NAME = 'payment-service'
        SONAR_HOST = 'https://sonarqube.internal.company.com'
    }
    
    parameters {
        string(name: 'GIT_BRANCH', defaultValue: 'main', description: 'Branch to build')
        string(name: 'VERSION', defaultValue: '', description: 'Version override (leave empty for auto)')
        booleanParam(name: 'RUN_E2E_TESTS', defaultValue: true, description: 'Run end-to-end tests')
        booleanParam(name: 'PUSH_TO_REGISTRY', defaultValue: true, description: 'Push image to registry')
        choice(name: 'DEPLOY_ENV', choices: ['staging', 'production'], description: 'Deployment target')
    }
    
    stages {
        stage('Checkout') {
            agent {
                docker {
                    image 'alpine/git:2.45.1'
                    args '-u root:root --network=host -v /tmp/git-cache:/git-cache'
                }
            }
            steps {
                script {
                    checkout([
                        $class: 'GitSCM',
                        branches: [[name: "refs/heads/${params.GIT_BRANCH}"]],
                        userRemoteConfigs: [[
                            url: 'git@github.com:myorg/payment-service.git',
                            credentialsId: 'github-app-token'
                        ]],
                        extensions: [
                            [$class: 'CleanCheckout'],
                            [$class: 'PruneStaleBranch'],
                            [$class: 'SubmoduleOption', disableSubmodules: true]
                        ]
                    ])
                    
                    GIT_COMMIT = sh(
                        script: 'git rev-parse HEAD',
                        returnStdout: true
                    ).trim()
                    
                    GIT_SHORT_COMMIT = GIT_COMMIT.take(7)
                    
                    if (params.VERSION) {
                        APP_VERSION = params.VERSION
                    } else {
                        APP_VERSION = sh(
                            script: 'git describe --tags --always --dirty 2>/dev/null || echo "dev-${GIT_SHORT_COMMIT}"',
                            returnStdout: true
                        ).trim()
                    }
                }
            }
        }
        
        stage('Dependency Scan') {
            agent {
                docker {
                    image 'owasp/dependency-check:9.0.9'
                    args '--network=none -e NVD_API_KEY=${NVD_API_KEY}'
                }
            }
            steps {
                dir('payment-service') {
                    sh '''
                        dependency-check.sh \
                            --scan . \
                            --format JSON \
                            --out dependency-check-report.json \
                            --failOnCVSS 7 \
                            --nvdApiKey "${NVD_API_KEY}" \
                            --suppression suppression.xml
                    '''
                }
                archiveArtifacts artifacts: 'payment-service/dependency-check-report.json', fingerprint: true
            }
        }
        
        stage('Static Analysis') {
            agent {
                docker {
                    image 'sonarsource/sonar-scanner-cli:5.0.1'
                    args '--network=host -e SONAR_HOST_URL=${SONAR_HOST}'
                }
            }
            steps {
                dir('payment-service') {
                    withSonarQubeEnv('sonarqube-production') {
                        sh '''
                            sonar-scanner \
                                -Dsonar.projectKey=payment-service \
                                -Dsonar.projectVersion=${APP_VERSION} \
                                -Dsonar.git.commit=${GIT_COMMIT} \
                                -Dsonar.qualitygate.wait=true \
                                -Dsonar.qualitygate.timeout=300
                        '''
                    }
                }
            }
        }
        
        stage('Build') {
            agent {
                docker {
                    image 'gradle:8.7.0-jdk17'
                    args '-v /var/jenkins_home/gradle-cache:/home/gradle/.gradle -v /var/run/docker.sock:/var/run/docker.sock:ro'
                    reuseNode true
                }
            }
            steps {
                dir('payment-service') {
                    sh '''
                        gradle clean build \
                            -Pversion=${APP_VERSION} \
                            --no-daemon \
                            --parallel \
                            --build-cache \
                            -x integrationTest
                    '''
                }
                stash includes: 'payment-service/build/libs/*.jar', name: 'artifacts'
            }
        }
        
        stage('Unit Tests') {
            agent {
                docker {
                    image 'gradle:8.7.0-jdk17'
                    args '-v /var/jenkins_home/gradle-cache:/home/gradle/.gradle'
                    reuseNode true
                }
            }
            steps {
                dir('payment-service') {
                    sh 'gradle test --no-daemon'
                }
            }
            post {
                always {
                    junit 'payment-service/build/test-results/**/*.xml'
                    jacoco(
                        execPattern: 'payment-service/build/jacoco/**/*.exec',
                        classPattern: 'payment-service/build/classes',
                        sourcePattern: 'payment-service/src/main/java'
                    )
                }
            }
        }
        
        stage('Docker Build') {
            agent {
                docker {
                    image 'docker:27.3.1-dind'
                    args '--privileged --shm-size=2g -v /var/jenkins_home/docker-cache:/root/.docker'
                }
            }
            steps {
                script {
                    unstash 'artifacts'
                    
                    def dockerfile = 'payment-service/Dockerfile'
                    def imageTag = "${DOCKER_REGISTRY}/${DOCKER_ORG}/${APP_NAME}:${APP_VERSION}"
                    def imageTagLatest = "${DOCKER_REGISTRY}/${DOCKER_ORG}/${APP_NAME}:latest"
                    def imageTagCommit = "${DOCKER_REGISTRY}/${DOCKER_ORG}/${APP_NAME}:${GIT_SHORT_COMMIT}"
                    
                    sh """
                        docker build \
                            --file ${dockerfile} \
                            --tag ${imageTag} \
                            --tag ${imageTagCommit} \
                            --build-arg APP_VERSION=${APP_VERSION} \
                            --build-arg GIT_COMMIT=${GIT_COMMIT} \
                            --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
                            --label "org.opencontainers.image.version=${APP_VERSION}" \
                            --label "org.opencontainers.image.revision=${GIT_COMMIT}" \
                            --label "org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
                            --label "build.jenkins.url=${BUILD_URL}" \
                            --no-cache \
                            payment-service/
                    """
                    
                    docker.withRegistry("https://${DOCKER_REGISTRY}", 'docker-registry-password') {
                        if (params.PUSH_TO_REGISTRY) {
                            sh "docker push ${imageTag}"
                            sh "docker push ${imageTagCommit}"
                            
                            if (params.GIT_BRANCH == 'main' && params.DEPLOY_ENV == 'production') {
                                sh "docker tag ${imageTag} ${imageTagLatest}"
                                sh "docker push ${imageTagLatest}"
                            }
                        }
                    }
                    
                    currentBuild.description = "Image: ${imageTag}"
                }
            }
        }
        
        stage('Container Security Scan') {
            agent {
                docker {
                    image 'aquasec/trivy:0.50.1'
                    args '--network=host -v /var/run/docker.sock:/var/run/docker.sock:ro'
                }
            }
            steps {
                sh """
                    trivy image \
                        --severity HIGH,CRITICAL \
                        --exit-code 1 \
                        --ignore-unfixed \
                        --format json \
                        --output trivy-report.json \
                        ${DOCKER_REGISTRY}/${DOCKER_ORG}/${APP_NAME}:${APP_VERSION}
                """
                archiveArtifacts artifacts: 'trivy-report.json', fingerprint: true
            }
        }
        
        stage('Integration Tests') {
            agent {
                docker {
                    image 'gradle:8.7.0-jdk17'
                    args '-v /var/jenkins_home/gradle-cache:/home/gradle/.gradle -e TEST_DB_HOST=database.internal.company.com'
                    reuseNode true
                }
            }
            steps {
                dir('payment-service') {
                    sh 'gradle integrationTest --no-daemon -Dtest.db.host=${TEST_DB_HOST}'
                }
            }
        }
        
        stage('E2E Tests') {
            when {
                expression { params.RUN_E2E_TESTS }
            }
            agent {
                docker {
                    image 'cypress/included:13.13.1'
                    args '--network=host -e CYPRESS_BASE_URL=https://staging.internal.company.com'
                }
            }
            steps {
                dir('payment-service/e2e') {
                    sh '''
                        export CYPRESS_SCREENSHOTS_FOLDER=/tmp/cypress-screenshots
                        cypress run \
                            --config video=true \
                            --config screenshotOnFailure=true \
                            --reporter junit \
                            --reporter-options "mochaFile=results/results-[hash].xml"
                    '''
                }
            }
            post {
                always {
                    junit 'payment-service/e2e/results/*.xml'
                    archiveArtifacts artifacts: 'payment-service/e2e/cypress/videos/**/*.mp4', fingerprint: true
                }
            }
        }
        
        stage('Deploy') {
            when {
                expression { params.GIT_BRANCH == 'main' && params.DEPLOY_ENV == 'production' }
            }
            agent {
                docker {
                    image 'hashicorp/terraform:1.9.6'
                    args '--network=host -v /var/jenkins_home/terraform-state:/terraform/state'
                }
            }
            steps {
                dir('payment-service/infrastructure') {
                    sh '''
                        terraform init \
                            -backend-config="bucket=production-tf-state" \
                            -backend-config="key=payment-service/terraform.tfstate"
                        
                        terraform plan \
                            -var="image_tag=${APP_VERSION}" \
                            -var="environment=production" \
                            -out=plan.tfplan
                        
                        terraform apply -auto-approve plan.tfplan
                    '''
                }
            }
        }
    }
    
    post {
        success {
            script {
                if (params.DEPLOY_ENV == 'production') {
                    slackSend(
                        channel: '#production-deployments',
                        color: 'good',
                        message: "✅ Deployment successful: ${APP_NAME} ${APP_VERSION} deployed to production by ${env.BUILD_USER_ID} - ${BUILD_URL}"
                    )
                }
            }
        }
        failure {
            script {
                slackSend(
                    channel: '#ci-alerts',
                    color: 'danger',
                    message: "❌ Build failed: ${APP_NAME} ${APP_VERSION} - Stage: ${env.FAILED_STAGE} - ${BUILD_URL}"
                )
            }
        }
        always {
            cleanWs(
                cleanWhenNotBuilt: false,
                deleteDirs: true,
                disableDeferredWipeout: true,
                notFailBuild: true
            )
            script {
                sh 'docker system prune -f --filter "until=24h" --volumes || true'
            }
        }
    }
}

This pipeline demonstrates production patterns: each stage uses a purpose-built Docker image, sensitive tools run with --network=none, artifacts are archived for audit trails, security scanning gates block deployment on HIGH or CRITICAL vulnerabilities, and Terraform handles infrastructure changes. The reuseNode directive preserves the workspace between stages that need shared state.

Production Hardening Best Practices

1. Secrets Management Never in Plaintext

Production Jenkins never stores secrets in job configurations or pipeline code. Instead, secrets flow through a chain: external secret store → environment variable → Jenkins credential store → pipeline injection:

# Vault policy for Jenkins master
# jenkins-master-policy.hcl
path "secret/data/ci/*" {
  capabilities = ["read", "list"]
}

path "secret/data/production/deploy-keys/*" {
  capabilities = ["read"]
}

# Never grant write or delete to CI systems
path "secret/*" {
  capabilities = ["deny"]
}

# Jenkins master retrieves secrets at startup, never during builds
# Builds get secrets via Jenkins credential binding, not direct Vault access

The principle is: the Jenkins master fetches secrets at startup for its own configuration. Build jobs never have direct access to Vault—they receive secrets through Jenkins' credential binding mechanism, which injects them as masked environment variables scoped to a single stage.

2. Docker Socket Security

Mounting /var/run/docker.sock gives the container root-equivalent access to the host. In production, mitigate this with:

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo