← Back to DevBytes
Git July 09, 2026 13 min read

Fix Git 'Permission denied (publickey)' Error in Production: Root Cause Analysis

Understanding the 'Permission denied (publickey)' Error

Few error messages in a developer's workflow are as immediately panic-inducing as Permission denied (publickey) when trying to push code or pull dependencies in a production environment. This error signals that SSH authentication has failed between your machine and the remote Git server. In production, where deployments are time-sensitive and often automated, this failure can halt pipelines, block releases, and introduce costly downtime while teams scramble for a fix.

The error message typically appears in one of two forms:

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Or, when running in a script or CI/CD pipeline:

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

At its core, the error means the SSH client attempted to authenticate using a key that the remote server does not recognize—or that no key was offered at all. In production systems, this is rarely a simple oversight. The root cause often lurks in environment drift, agent forwarding gaps, orphaned processes, or subtle filesystem permission issues that only manifest under the specific conditions of a deployed environment.

Why This Error Is Critical in Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

In local development, a publickey denial is an inconvenience—you regenerate a key, add it to your SSH agent, and move on. In production, the stakes are fundamentally different:

Understanding the why behind this error in production—not just the quick fix—is what separates reactive firefighting from mature operational practice.

Root Cause Analysis: The Five Most Common Production Scenarios

Scenario 1: SSH Agent Forwarding Is Not Configured

Many teams rely on SSH agent forwarding to propagate credentials from a developer's machine through a bastion host to the final production server. When agent forwarding is not enabled or the chain is broken at any hop, the target server has no access to the private key.

How it happens: A developer SSHes into a production jump host but forgets the -A flag, or the SSH daemon on the intermediate server has agent forwarding disabled via AllowAgentForwarding no. From the jump host, any Git operation requiring the developer's key fails.

Detection:

# On the production server, check if the agent socket exists
echo $SSH_AUTH_SOCK

# If empty, agent forwarding is not active
# Also verify with:
ssh-add -l
# Output: Could not open a connection to your authentication agent.

Scenario 2: The SSH Key File Has Incorrect Permissions

SSH is deliberately strict about file permissions on private keys. If a private key file is readable by group or world, SSH refuses to use it—period. In production, automated provisioning scripts or configuration management tools sometimes create key files with overly permissive modes, or a hurried operator runs chmod incorrectly.

Required permissions:

Detection and fix:

# Check current permissions
ls -la ~/.ssh/
# Look for anything other than drwx------ for .ssh and -rw------- for key files

# Fix permissions immediately
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
chmod 600 ~/.ssh/authorized_keys 2>/dev/null || true

Even a single world-readable bit on the private key will cause SSH to emit the dreaded publickey denial and refuse to proceed.

Scenario 3: The SSH Agent Has Lost the Key (Or Never Had It)

In long-running production sessions, SSH agents can expire, crash, or get cleared. This is especially common in containerized environments where the agent process might be running in a different namespace or where the agent socket was not mounted into the container.

How it manifests: A script that worked perfectly during the morning deployment suddenly fails in the evening. The agent process is still alive, but the key was never added, or was added with a time-limited expiration that has lapsed.

Detection:

# List keys currently held by the agent
ssh-add -l
# Output: The agent has no identities.

# Or, if the agent socket is missing entirely:
echo $SSH_AUTH_SOCK
# Empty output means no agent connection

Immediate remediation:

# Start the agent if needed
eval "$(ssh-agent -s)"

# Add the key (you'll need the passphrase if applicable)
ssh-add ~/.ssh/id_ed25519

# Verify
ssh-add -l
# Should show the key fingerprint

Scenario 4: The Key Has Been Regenerated or Rotated Server-Side

In environments with key rotation policies, the public key registered on the Git server (GitHub, GitLab, Bitbucket, or a self-hosted Gitea instance) may have been replaced while the private key on the production server remains the old one. This creates a silent mismatch: the key exists, permissions are correct, the agent is running—but authentication fails because the server no longer trusts the offered key.

Detection requires server-side comparison:

# Derive the fingerprint of the local private key
ssh-keygen -lf ~/.ssh/id_ed25519

# Compare this fingerprint to what is registered on the Git server
# For GitHub, check: https://github.com/settings/keys
# The fingerprints must match exactly

This scenario is notoriously difficult to debug because all local indicators suggest everything is fine. The fix is to register the new public key on the Git server or replace the local private key with the one corresponding to the registered public key.

Scenario 5: The Production Environment Uses a Different User Context

The most insidious production-only cause: the SSH key is correctly configured for the deploy user, but the Git command is being executed by a different user—often root, www-data, or a container's default user. Each user has its own ~/.ssh directory, and the key in /home/deploy/.ssh is invisible to a process running as www-data.

Common triggers:

Detection:

# Determine who is actually running the Git command
whoami
id

# Check which SSH configuration is being used
# SSH looks in the home directory of the current user
echo ~
ls -la ~/.ssh/

Fix approaches:

Systematic Debugging: A Step-by-Step Workflow

When you encounter the error in production, resist the urge to blindly try fixes. Follow this structured diagnostic sequence to identify the true root cause before applying a remedy.

Step 1: Enable Verbose SSH Output

SSH's verbose mode is your most powerful diagnostic tool. The -v flag (or -vvv for maximum detail) reveals exactly which keys are offered, which authentication methods are attempted, and where the chain breaks.

# Test SSH connectivity to the Git server with verbose output
ssh -vT git@github.com

# For even more detail, especially around key offering:
ssh -vvvT git@github.com 2>&1 | grep -E "Offering|Authentication|publickey|denied"

Key lines to look for in the verbose output:

debug1: Offering public key: /home/deploy/.ssh/id_ed25519
debug1: Server accepts key: /home/deploy/.ssh/id_ed25519
# Success case

debug1: Offering public key: /home/deploy/.ssh/id_ed25519
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
# Failure case — the server rejected the offered key

If you see no "Offering public key" lines at all, the key was never loaded by the agent or found by SSH. If you see offerings followed by rejection, the public key on the server does not match.

Step 2: Verify the SSH Agent State

# Check if agent is running and socket is accessible
ps aux | grep ssh-agent
echo $SSH_AUTH_SOCK

# List identities the agent holds
ssh-add -l

# If needed, explicitly add the key with debugging
ssh-add -L  # Lists public keys the agent can offer

Step 3: Inspect Key Files and Permissions Recursively

# Full audit of the .ssh directory
find ~/.ssh -ls

# Expected output pattern:
# drwx------  deploy deploy .ssh/
# -rw-------  deploy deploy .ssh/id_ed25519
# -rw-r--r--  deploy deploy .ssh/id_ed25519.pub
# -rw-------  deploy deploy .ssh/authorized_keys

Step 4: Compare Local and Remote Public Keys

# Get the fingerprint and comment of the local key
ssh-keygen -lf ~/.ssh/id_ed25519

# Extract the public key to compare with what's registered
cat ~/.ssh/id_ed25519.pub

# For GitHub, use the API to list registered keys
# (requires a working access token if SSH is broken)
curl -H "Authorization: token YOUR_TOKEN" https://api.github.com/user/keys

Step 5: Check for Host Key or SSH Config Mismatches

Sometimes the issue is not the user key but a mismatch in known_hosts or SSH configuration that causes the client to connect to an unexpected endpoint.

# Verify the SSH configuration being used
cat ~/.ssh/config

# Check known_hosts for the Git server
ssh-keygen -F github.com

# Test with explicit host key checking disabled (for diagnosis only)
ssh -o StrictHostKeyChecking=no -T git@github.com

Production-Ready Solutions

Solution A: Deploy Keys — The Gold Standard for Production

Deploy keys are SSH keys scoped to a single repository rather than a user account. They are the recommended approach for production servers because they follow the principle of least privilege: a compromised key grants access to exactly one repository, not every repository the owning user can access.

Setting up a deploy key:

# Generate a dedicated key pair on the production server
ssh-keygen -t ed25519 -C "production-deploy-$(hostname)" -f ~/.ssh/deploy_key_production

# The public key to register:
cat ~/.ssh/deploy_key_production.pub

Register this public key in your Git hosting platform (GitHub: Repository → Settings → Deploy Keys; GitLab: Repository → Settings → Repository → Deploy Keys). Grant read-only access unless write access is specifically required.

Configure SSH to use the deploy key for Git operations:

# Add to ~/.ssh/config
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/deploy_key_production
    IdentitiesOnly yes
    StrictHostKeyChecking yes

The IdentitiesOnly yes directive is critical: it tells SSH to use only the specified key and not fall back to any other keys the agent might hold, preventing accidental authentication with a wrong key.

Solution B: SSH Agent Forwarding with Strict Controls

When deploy keys are not feasible—for example, when a production server needs access to multiple repositories tied to different users—agent forwarding can be used, but it must be configured with tight constraints.

# On the developer's machine, connect with agent forwarding
ssh -A user@production-server

# On the production server, verify the agent is accessible
ssh-add -l

# For automated scripts, ensure the agent socket persists
export SSH_AUTH_SOCK=$(find /tmp -name "agent.*" -user $(whoami) 2>/dev/null | head -1)

Security note: Agent forwarding exposes the agent socket to anyone with root access on the remote server. For high-security environments, consider using ProxyJump with key-wrapped SSH certificates instead.

Solution C: Environment-Specific SSH Configuration

For production environments where multiple services run under different user accounts, create a shared SSH configuration that each service can reference.

# Create a system-wide configuration snippet
# /etc/ssh/ssh_config.d/production-git.conf

Host github.com
    User git
    IdentityFile /etc/ssh/keys/deploy_rsa
    IdentitiesOnly yes

Host gitlab.internal
    User git
    IdentityFile /etc/ssh/keys/gitlab_deploy_rsa
    IdentitiesOnly yes

Ensure the key file permissions are locked down:

chmod 600 /etc/ssh/keys/deploy_rsa
chown root:root /etc/ssh/keys/deploy_rsa

Then, in your deployment scripts or CI/CD configuration, explicitly point to this configuration:

# Use the system SSH config
export GIT_SSH_COMMAND="ssh -F /etc/ssh/ssh_config.d/production-git.conf"
git clone git@github.com:organization/repo.git

Solution D: Container-Native SSH Handling

In Docker-based production environments, SSH keys must be deliberately mounted or injected. Never bake private keys into container images—they become immortalized in image layers.

Using Docker BuildKit secrets (for build-time Git operations):

# Dockerfile
FROM alpine:latest
RUN --mount=type=ssh,id=github \
    git clone git@github.com:organization/private-repo.git /app

# Build command
docker build --ssh default=~/.ssh/id_ed25519 -t myapp .

For runtime Git operations in containers, mount the SSH directory as a volume:

# docker-compose.yml
services:
  app:
    image: myapp:latest
    volumes:
      - /home/deploy/.ssh:/home/appuser/.ssh:ro
    environment:
      - GIT_SSH_COMMAND=ssh -o StrictHostKeyChecking=yes

Alternative for Kubernetes: Use a Secret:

# Create the secret from existing keys
kubectl create secret generic git-ssh-key \
  --from-file=ssh-privatekey=/home/deploy/.ssh/id_ed25519 \
  --from-file=ssh-publickey=/home/deploy/.ssh/id_ed25519.pub

# Mount in the pod specification
volumes:
  - name: git-ssh
    secret:
      secretName: git-ssh-key
      defaultMode: 0600
containers:
  - name: app
    volumeMounts:
      - name: git-ssh
        mountPath: /root/.ssh
        readOnly: true

Preventive Measures and Best Practices

1. Implement SSH Health Checks in Deployment Pipelines

Before any Git-dependent step runs, validate SSH connectivity as a discrete pipeline stage:

# SSH health check script for CI/CD
#!/bin/bash
set -e

echo "=== SSH Health Check ==="

# Check agent
if [ -n "$SSH_AUTH_SOCK" ]; then
  echo "SSH agent socket: $SSH_AUTH_SOCK"
  ssh-add -l || echo "WARNING: Agent has no identities"
else
  echo "No SSH agent detected"
fi

# Check key file permissions
find ~/.ssh -type f -name "id_*" ! -name "*.pub" -exec stat -c "%a %n" {} \; | while read perm file; do
  if [ "$perm" != "600" ]; then
    echo "ERROR: $file has permissions $perm (requires 600)"
    exit 1
  fi
done

# Test connection
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new -T git@github.com 2>&1 | grep -q "successfully authenticated" && \
  echo "SSH connection to GitHub: OK" || \
  echo "SSH connection to GitHub: FAILED"

2. Use SSH Certificates Instead of Raw Keys (Advanced)

For larger production fleets, SSH certificates signed by a trusted CA eliminate the need to distribute and rotate individual keys across servers. Each server gets a short-lived certificate that automatically expires.

# Generate a certificate (simplified example with Vault)
vault write -field=signed_key ssh-client-signer/sign/my-role \
  public_key=@/home/deploy/.ssh/id_ed25519.pub > ~/.ssh/id_ed25519-cert.pub

# SSH automatically uses the certificate alongside the private key
ssh -T git@github.com
# The certificate is offered and validated by the server

3. Monitor Key Expiration and Rotation

Embed key metadata checks into your monitoring system:

# Check when a key was created
ssh-keygen -lf ~/.ssh/id_ed25519 | awk '{print $1, $4}'

# Script to alert on keys older than 90 days
#!/bin/bash
KEY_AGE_DAYS=$(( ($(date +%s) - $(date -r ~/.ssh/id_ed25519 +%s)) / 86400 ))
if [ $KEY_AGE_DAYS -gt 90 ]; then
  echo "ALERT: SSH key is $KEY_AGE_DAYS days old — rotation required"
fi

4. Document the SSH Chain of Trust

Maintain a living diagram or runbook that maps exactly which keys exist on which servers, which public keys are registered where, and who is responsible for rotation. When the publickey error strikes at 2 AM, this documentation prevents the scramble.

5. Avoid Common Anti-Patterns

Emergency Recovery: When Nothing Works

In a true production outage where SSH-based Git access is completely blocked and you cannot immediately resolve the key issue, you have a temporary escape hatch: fall back to HTTPS with a personal access token (PAT) or an OAuth token. This bypasses SSH entirely.

# Temporary override using HTTPS and a token
git remote set-url origin https://github.com/organization/repo.git
git config --global url."https://${GIT_TOKEN}:x-oauth-basic@github.com/".insteadOf "https://github.com/"
git pull origin main

# After the SSH issue is resolved, revert
git remote set-url origin git@github.com:organization/repo.git
git config --global --unset url."https://${GIT_TOKEN}:x-oauth-basic@github.com/".insteadOf

This is not a permanent solution—it introduces token management complexity and the risk of token leakage—but it can restore service while the root cause of the SSH failure is addressed.

Conclusion

The Permission denied (publickey) error in production is never just a key problem—it is a symptom of a breakdown in your SSH trust chain. Whether the root cause is agent forwarding gone silent, a permissions drift introduced by a configuration management run, a user context mismatch in a containerized deployment, or an expired key that slipped past rotation monitoring, the fix lies in systematic diagnosis rather than reflexive key regeneration.

By implementing deploy keys scoped to single repositories, baking SSH health checks into your CI/CD pipelines, enforcing strict file permissions through infrastructure-as-code, and maintaining clear documentation of your SSH trust architecture, you transform this error from a recurring crisis into a rare and quickly resolvable anomaly. The investment in understanding the root cause mechanisms pays dividends every time a production deployment proceeds smoothly—and every time an on-call engineer can diagnose the issue in seconds rather than hours.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles