Fix 'pip install fails' with SSL Certificate Error in Production: Root Cause Analysis

What is the pip SSL Certificate Error?

When you run pip install in a production environment, you may encounter an error that looks like this:

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

This error occurs when Python's pip cannot verify the SSL certificate presented by PyPI (the Python Package Index) or a custom package repository. The underlying TLS handshake fails because the client (pip) cannot validate the certificate chain presented by the server. This is a critical security mechanism designed to prevent man-in-the-middle attacks, but in production environments with specific configurations—corporate proxies, custom CA certificates, air-gapped networks, or outdated certificate bundles—it can become a blocking issue.

Root Cause Analysis: Why SSL Certificate Errors Occur in Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Understanding the root cause is essential before applying a fix. The error stems from one or more of the following scenarios:

1. Missing or Outdated Root CA Certificates

Python relies on a bundled certificate store provided by the certifi package or the system's OpenSSL CA bundle. In minimal production Docker containers or stripped-down server images, the CA certificates package may not be installed at all, or the certifi version bundled with Python may be severely outdated.

2. Corporate TLS Interception Proxies (MITM)

Many enterprise production environments deploy TLS inspection proxies that terminate and re-sign HTTPS traffic using an internal Certificate Authority (CA). This CA certificate is not part of the public trust chain, so pip—which uses the public CA bundle—rejects the connection. The proxy acts as a man-in-the-middle by design, and pip correctly identifies the certificate as untrusted.

3. Custom Package Repositories with Self-Signed Certificates

Production environments often use private PyPI mirrors (like Artifactory, Nexus, or AWS CodeArtifact) that may use self-signed or internal CA-signed certificates. Pip has no way to validate these certificates against the default trust store.

4. Python Compiled Without SSL Support or Using a Different OpenSSL Version

In some cases, Python is compiled against a specific OpenSSL library version, but the CA bundle path is hard-coded to a location that does not exist or is empty. This is common in custom-built Python installations or when using multiple Python versions installed via pyenv or manual compilation.

5. System Clock Skew

SSL certificate validity depends on accurate system time. If the server's clock is significantly off (wrong timezone, NTP not configured), certificates may appear expired or not-yet-valid, causing verification failures.

Why This Matters in Production

SSL certificate errors in production are not just a nuisance—they represent a serious operational risk:

Blocked CI/CD pipelines: Automated builds that fetch dependencies during container image construction will fail, halting deployments.
Security compliance violations: Workarounds like disabling SSL verification (--trusted-host or CERT_REQUIRED=0) bypass critical security controls and may violate PCI-DSS, SOC2, or internal security policies.
Supply chain integrity: Without proper certificate validation, you cannot guarantee that packages are fetched from the genuine repository and not tampered with.
Audit findings: Security auditors actively flag disabled SSL verification as a high-severity finding.

Therefore, the fix must preserve SSL verification rather than disabling it.

How to Diagnose the Exact Root Cause

Before applying any fix, systematically diagnose the problem:

Step 1: Reproduce the Error with Verbose Output

pip install --verbose --no-cache-dir requests 2>&1 | grep -i ssl

This shows the exact SSL context and certificate chain information pip is using.

Step 2: Test SSL Connectivity Independently

openssl s_client -connect pypi.org:443 -showcerts

If this succeeds but pip fails, the issue is specific to Python's certificate bundle, not the system trust store.

Step 3: Check Python's Certificate Bundle Location

python -c "import certifi; print(certifi.where())"
python -c "import ssl; print(ssl.get_default_verify_paths())"

This reveals which CA bundle file Python is actually using. Verify that the file exists and contains valid certificates.

Step 4: Identify Interception Proxy Headers

curl -v https://pypi.org/simple/ 2>&1 | grep -i "issuer\|subject\|proxy"

If the issuer CN (Common Name) contains your company's name rather than a public CA, you have an interception proxy.

Production-Ready Solutions (That Preserve SSL Verification)

Solution 1: Install CA Certificates in the Container/Server (The Standard Fix)

For missing system CA bundles, install the appropriate package during the image build:

Debian/Ubuntu Dockerfile:

RUN apt-get update && apt-get install -y ca-certificates && update-ca-certificates

Alpine Linux Dockerfile (critical—often missed):

RUN apk add --no-cache ca-certificates openssl

Red Hat/CentOS/Rocky:

RUN yum install -y ca-certificates && update-ca-trust

After installation, verify pip works immediately:

pip install --no-cache-dir requests

Solution 2: Append Corporate CA to Python's Trust Store

When a corporate proxy re-signs traffic, add the internal CA certificate to the bundle that Python uses. This preserves full validation.

Step 1: Obtain the corporate CA certificate in PEM format. Often available via internal documentation or by exporting from a browser that trusts it.

# Save the corporate CA cert to a file
echo "-----BEGIN CERTIFICATE-----
...corporate CA cert content...
-----END CERTIFICATE-----" > /usr/local/share/ca-certificates/corp-ca.crt

Step 2: Append it to the certifi bundle or update the system store:

# Option A: Append to certifi's bundle directly
python -c "
import certifi, shutil
bundle = certifi.where()
with open(bundle, 'a') as f:
    with open('/usr/local/share/ca-certificates/corp-ca.crt') as ca:
        f.write(ca.read())
print('Appended corporate CA to certifi bundle')
"

Option B (preferred): Update the system trust store and point pip at it:

# Debian/Ubuntu
cp /usr/local/share/ca-certificates/corp-ca.crt /usr/local/share/ca-certificates/
update-ca-certificates

# Then use the system bundle with pip
pip install --cert /etc/ssl/certs/ca-certificates.crt requests

Step 3: Make it permanent via environment variable (production standard):

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

Add these to your container's environment or /etc/environment so all Python HTTP clients (not just pip) trust the corporate CA.

Solution 3: Configure pip.conf with a Custom Certificate Bundle

For teams that cannot modify the system trust store, configure pip specifically:

# /etc/pip.conf or ~/.pip/pip.conf
[global]
cert = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
# Or for a specific custom repository
[global]
index-url = https://artifactory.internal.company.com/artifactory/api/pypi/pypi/simple
cert = /opt/certs/internal-ca-bundle.pem

This approach is ideal for Docker builds because you can copy a CA file and reference it cleanly:

COPY internal-ca-bundle.pem /etc/pip-certs/internal-ca-bundle.pem
COPY pip.conf /etc/pip.conf
RUN pip install --no-cache-dir -r requirements.txt

Solution 4: Use the REQUESTS_CA_BUNDLE Environment Variable (Requests Library)

Since pip uses the requests library internally, the REQUESTS_CA_BUNDLE environment variable overrides the CA bundle path for all HTTP requests made by pip:

ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
RUN pip install --no-cache-dir -r requirements.txt

This is a clean, non-invasive solution that works across all tools using the requests library in the same container.

Solution 5: Use a PyPI Mirror with Proper Public Certificates

If the root cause is a self-signed internal mirror, either:

Provision a proper publicly-trusted certificate for the internal mirror (via Let's Encrypt or internal ACME server)
Use a cloud-based private PyPI service that automatically provides valid TLS (AWS CodeArtifact, Azure Artifacts, Google Artifact Registry)

# AWS CodeArtifact example (always has valid public TLS)
pip install \
  --index-url https://aws:codeartifact@my-domain-1234567890.d.codeartifact.region.amazonaws.com/v3/python/my-repo/simple/ \
  --no-cache-dir requests

Solution 6: Fix System Clock Issues

Ensure NTP synchronization is working:

# Check current time and NTP status
timedatectl status
ntpq -p

# Force sync (systemd-based systems)
timedatectl set-ntp true
systemctl restart systemd-timesyncd

# For containers, ensure the host's time is correct—containers inherit host time
# Or run ntpd inside the container if necessary

What NOT to Do: Dangerous Anti-Patterns

The following approaches are commonly suggested online but create security vulnerabilities:

DO NOT use --trusted-host:

# INSECURE - bypasses SSL verification entirely
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org package-name

DO NOT set PIP_TRUSTED_HOST environment variable:

# INSECURE
ENV PIP_TRUSTED_HOST="pypi.org files.pythonhosted.org"

DO NOT disable SSL globally in pip.conf:

# INSECURE - never do this in production
[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org

DO NOT monkey-patch SSL verification in Python:

# INSECURE - this is a red flag for any security audit
import ssl
ssl._create_default_https_context = ssl._create_unverified_context

These workarounds will allow pip to download packages, but they also allow any attacker who can perform DNS spoofing or network interception to inject malicious packages into your build pipeline.

Production Dockerfile Template: The Definitive Pattern

Here is a complete, production-grade Dockerfile pattern that handles SSL certificates correctly across different base images:

# Stage 1: Builder
FROM python:3.12-slim AS builder

# Install system CA certificates (critical)
RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates && \
    update-ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# Copy corporate CA certificate if using TLS inspection proxy
# COPY corp-ca-cert.pem /usr/local/share/ca-certificates/corp-ca-cert.crt
# RUN update-ca-certificates

# Set CA bundle environment variables
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
    SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

# Copy and install dependencies with full SSL verification
COPY requirements.txt .
RUN pip install --no-cache-dir --require-hashes -r requirements.txt

# Stage 2: Runtime
FROM python:3.12-slim

RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates && \
    update-ca-certificates && \
    rm -rf /var/lib/apt/lists/*

ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
    SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY app/ /app/

WORKDIR /app
USER 1000:1000
CMD ["python", "main.py"]

This pattern ensures:

CA certificates are always present in both build and runtime stages
The environment variables are set so all Python HTTP libraries use the system trust store
No SSL verification is ever disabled
Multi-stage builds keep the final image small while maintaining security

Best Practices for Production SSL Certificate Management with pip

Always install ca-certificates in Docker images: This is the most overlooked step when using slim or Alpine base images. Make it part of your base image template.
Set REQUESTS_CA_BUNDLE and SSL_CERT_FILE as environment variables: This ensures consistency across pip, requests, urllib, and any other HTTP library that respects these standard variables.
Use --require-hashes in production requirements files: This provides defense in depth—even if SSL is somehow bypassed, package integrity is verified via cryptographic hashes.
Pin the certifi package version: Include certifi in your requirements.txt with a pinned version to ensure the CA bundle is up to date.
Test SSL verification in CI: Add a step that explicitly verifies pip can connect to your package repositories with SSL validation enabled. A failure should block the build.
Document corporate CA requirements in your runbooks: If your organization uses TLS inspection, document where to obtain the CA certificate and how to configure it for Python applications.
Never commit CA certificates to public repositories: Store them in a secrets manager (HashiCorp Vault, AWS Secrets Manager) or a secure internal distribution mechanism.
Monitor for SSL-related errors in production logs: Set up alerting for CERTIFICATE_VERIFY_FAILED errors—they often indicate an expired internal CA certificate or a configuration drift.

Automated Validation Script

Use this script in your CI/CD pipeline to verify SSL is working correctly before proceeding with builds:

#!/bin/bash
# ssl-verify-check.sh - Fail build if SSL is broken or disabled

set -euo pipefail

echo "=== Testing pip SSL verification ==="

# Test 1: Verify pip can fetch from PyPI with SSL
if ! pip install --no-cache-dir --dry-run requests > /dev/null 2>&1; then
    echo "FAIL: pip cannot connect to PyPI with SSL verification"
    exit 1
fi

# Test 2: Verify no insecure configurations exist
if grep -r "trusted-host" /etc/pip.conf ~/.pip/pip.conf 2>/dev/null; then
    echo "FAIL: Insecure trusted-host configuration detected"
    exit 1
fi

# Test 3: Verify CA bundle file exists and is non-empty
CA_BUNDLE=$(python -c "import certifi; print(certifi.where())")
if [ ! -s "$CA_BUNDLE" ]; then
    echo "FAIL: CA bundle file is missing or empty: $CA_BUNDLE"
    exit 1
fi

# Test 4: Verify no monkey-patching of SSL in source code
if grep -r "_create_unverified_context\|CERT_REQUIRED.*=.*0" . --include="*.py" 2>/dev/null; then
    echo "FAIL: SSL verification bypass detected in source code"
    exit 1
fi

echo "PASS: All SSL verification checks passed"

Handling Air-Gapped (Offline) Production Environments

In completely air-gapped environments where no external internet access exists, SSL certificate errors may still occur when connecting to internal mirrors. The same principles apply:

Ensure your internal PyPI mirror uses a certificate signed by an internal CA
Install the internal CA certificate in all containers and servers
Configure pip to use the internal mirror and trust the internal CA

# For air-gapped environments
RUN pip config set global.index-url https://internal-pypi.corp.local/simple/
RUN pip config set global.cert /etc/pki/ca-trust/source/anchors/internal-ca.pem
RUN pip install --no-cache-dir -r requirements.txt

Never resort to --trusted-host even in air-gapped networks. Internal networks can still be compromised, and lateral movement by an attacker could exploit disabled SSL verification.

Conclusion

SSL certificate errors with pip in production environments are almost always a symptom of missing or misconfigured trust stores, not a problem with pip itself. The root cause typically traces back to minimal container images lacking CA certificates, corporate TLS inspection proxies introducing untrusted internal CAs, or self-signed certificates on internal package repositories. The correct fix is always to add the appropriate CA certificates to the trust store rather than disabling SSL verification. By installing the ca-certificates package, appending corporate CAs to the certifi bundle, setting the REQUESTS_CA_BUNDLE environment variable, and validating SSL in CI pipelines, you can maintain both operational reliability and security compliance. The solutions presented here preserve end-to-end cryptographic verification of package integrity, ensuring your production deployments remain secure against supply chain attacks while eliminating the frustrating SSL certificate errors that block builds.