How to Host Multiple AI Agents on One Server with Docker

Introduction: What It Means to Host Multiple AI Agents on One Server

Hosting multiple AI agents on a single server means running several independent artificial intelligence services—such as language models, image generators, recommendation engines, or custom inference pipelines—on the same physical or virtual machine, each encapsulated in its own Docker container. Docker provides lightweight, isolated environments that package an agent's code, dependencies, and runtime into a portable unit. When you combine multiple agents under one roof using container orchestration tools like Docker Compose, you create a microservices-style architecture where each agent can be developed, scaled, and updated independently while sharing the host's CPU, RAM, storage, and GPU resources.

This tutorial walks you through the complete process: from understanding why this approach is valuable, to building a working multi-agent deployment with practical code examples, networking configurations, GPU sharing, and production-ready best practices.

Why Host Multiple AI Agents with Docker?

Running several AI agents on a single server with Docker offers compelling advantages over traditional monolithic deployments or managing separate virtual machines:

• Resource Efficiency: Containers share the host kernel and can be densely packed onto one machine, maximizing utilization of expensive hardware—especially GPU-equipped servers where every idle cycle costs money.
• Isolation and Reproducibility: Each agent runs with its own dependencies and Python environment (e.g., different PyTorch or TensorFlow versions) without conflicting with others. Docker images guarantee that the agent behaves identically in development, staging, and production.
• Simplified GPU Sharing: Multiple containers can access the same physical GPU(s) through Docker's device mapping, enabling several models to run inference concurrently without complex driver-level partitioning.
• Independent Scaling and Updates: You can restart, rebuild, or scale individual agents without affecting the rest. A bug fix in the summarization agent doesn't require redeploying the image captioning service.
• Unified Networking: Docker's internal networking lets agents communicate over a private bridge with service-name DNS resolution, making inter-agent calls as simple as HTTP requests to http://agent-b:8080.

Whether you're building an internal AI platform or a customer-facing product that chains multiple models together, Dockerizing each agent is the foundation for a clean, maintainable system.

Prerequisites

Before diving into the implementation, ensure your server has the following installed and configured:

• Docker Engine (version 24.0+) with the docker CLI
• Docker Compose (v2.x plugin or standalone binary)
• NVIDIA Container Toolkit (if using GPU acceleration — required for CUDA access inside containers)
• At least 16 GB RAM for comfortably running 2–3 medium-sized agents; more if models are large
• Python 3.10+ available on the host for building agent code (though containers will carry their own versions)

Verify GPU access with:

docker run --rm --gpus all nvidia/cuda:12.1-base nvidia-smi

If this command shows your GPU details, the NVIDIA Container Toolkit is correctly installed.

Architecture Overview

We will deploy three AI agents on a single server, each in its own container:

1. Summarizer Agent — a text summarization service using a Hugging Face transformer model, exposed on port 8001.
2. Image Generator Agent — a Stable Diffusion-based image creation service using Diffusers, exposed on port 8002 and requiring GPU access.
3. Chatbot Agent — a conversational agent backed by a quantized LLaMA model via llama-cpp-python, exposed on port 8003.

All three containers sit on a shared Docker network called agent_net. A lightweight NGINX reverse-proxy container routes external requests to the correct agent based on URL paths. Docker Compose manages the entire stack with a single configuration file.

The directory structure we'll build:

multi-agent-server/
├── docker-compose.yml
├── nginx/
│   └── default.conf
├── summarizer/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app.py
├── imagegen/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app.py
└── chatbot/
    ├── Dockerfile
    ├── requirements.txt
    └── app.py

Step-by-Step Implementation

1. Creating the Summarizer Agent

The summarizer agent uses the sshleifer/distilbart-cnn-12-6 model from Hugging Face, which is lightweight and runs well on CPU. Create the following files:

summarizer/requirements.txt:

transformers==4.36.2
torch==2.1.2
flask==3.0.0
gunicorn==21.2.0

summarizer/app.py:

from flask import Flask, request, jsonify
from transformers import pipeline
import torch

app = Flask(__name__)

# Load model once at startup
device = 0 if torch.cuda.is_available() else -1
summarizer_pipeline = pipeline(
    "summarization",
    model="sshleifer/distilbart-cnn-12-6",
    device=device
)

@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "ok", "device": "cuda" if device == 0 else "cpu"})

@app.route('/summarize', methods=['POST'])
def summarize():
    data = request.get_json()
    text = data.get("text", "")
    if not text:
        return jsonify({"error": "Missing 'text' field"}), 400
    
    max_length = data.get("max_length", 130)
    min_length = data.get("min_length", 30)
    
    result = summarizer_pipeline(
        text,
        max_length=max_length,
        min_length=min_length,
        do_sample=False
    )
    
    return jsonify({
        "summary": result[0]["summary_text"],
        "input_length": len(text.split()),
        "summary_length": len(result[0]["summary_text"].split())
    })

if __name__ == '__main__':
    # For development; production uses gunicorn via Dockerfile CMD
    app.run(host='0.0.0.0', port=8001)

summarizer/Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

# Pre-download the model during build to avoid cold-start latency
RUN python -c "from transformers import pipeline; pipeline('summarization', model='sshleifer/distilbart-cnn-12-6')"

EXPOSE 8001

CMD ["gunicorn", "--bind", "0.0.0.0:8001", "--workers", "2", "--timeout", "120", "app:app"]

Notice the RUN python -c ... layer — this downloads and caches the model at build time so the container starts ready to serve immediately.

2. Creating the Image Generator Agent (GPU-Accelerated)

This agent requires GPU access. We use the Diffusers library with Stable Diffusion. The Dockerfile will be built on a CUDA-capable base image.

imagegen/requirements.txt:

diffusers==0.25.0
transformers==4.36.2
accelerate==0.25.0
flask==3.0.0
gunicorn==21.2.0
pillow==10.2.0
torch==2.1.2

imagegen/app.py:

from flask import Flask, request, jsonify, send_file
from diffusers import StableDiffusionPipeline
import torch
import io
import base64

app = Flask(__name__)

device = "cuda" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if device == "cuda" else torch.float32

# Load Stable Diffusion pipeline
pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch_dtype,
    safety_checker=None  # Disable for simplicity in this tutorial
)
pipe = pipe.to(device)

@app.route('/health', methods=['GET'])
def health():
    return jsonify({
        "status": "ok",
        "device": device,
        "gpu_memory_allocated": torch.cuda.memory_allocated(0) if device == "cuda" else 0
    })

@app.route('/generate', methods=['POST'])
def generate():
    data = request.get_json()
    prompt = data.get("prompt", "A beautiful landscape")
    num_steps = data.get("steps", 25)
    guidance = data.get("guidance_scale", 7.5)
    
    try:
        with torch.no_grad():
            image = pipe(
                prompt=prompt,
                num_inference_steps=num_steps,
                guidance_scale=guidance
            ).images[0]
        
        # Convert PIL image to base64 for API response
        buf = io.BytesIO()
        image.save(buf, format="PNG")
        buf.seek(0)
        img_base64 = base64.b64encode(buf.read()).decode("utf-8")
        
        return jsonify({
            "prompt": prompt,
            "image_base64": img_base64,
            "format": "png"
        })
    except Exception as e:
        return jsonify({"error": str(e)}), 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8002)

imagegen/Dockerfile:

FROM nvidia/cuda:12.1-runtime-ubuntu22.04

# Install Python and pip
RUN apt-get update && apt-get install -y python3.11 python3-pip && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

COPY app.py .

# Pre-download the Stable Diffusion model
RUN python3 -c "from diffusers import StableDiffusionPipeline; \
    import torch; \
    pipe = StableDiffusionPipeline.from_pretrained('runwayml/stable-diffusion-v1-5', \
    torch_dtype=torch.float16); \
    print('Model cached successfully')"

EXPOSE 8002

CMD ["gunicorn", "--bind", "0.0.0.0:8002", "--workers", "1", "--timeout", "300", \
     "--worker-class", "sync", "app:app"]

Note the single gunicorn worker — GPU memory constraints often limit parallelism; one worker prevents multiple model copies from exhausting VRAM.

3. Creating the Chatbot Agent

This agent uses llama-cpp-python to run a quantized LLaMA model efficiently on CPU (or GPU if desired). We'll configure it for CPU inference to demonstrate resource diversity.

chatbot/requirements.txt:

llama-cpp-python==0.2.32
flask==3.0.0
gunicorn==21.2.0

chatbot/app.py:

from flask import Flask, request, jsonify
from llama_cpp import Llama
import os

app = Flask(__name__)

# Path to the GGUF model file — downloaded at build time
MODEL_PATH = os.environ.get("MODEL_PATH", "/app/models/llama-2-7b-chat.Q4_K_M.gguf")

llm = Llama(
    model_path=MODEL_PATH,
    n_ctx=2048,
    n_threads=8,
    verbose=False
)

@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "ok", "model_loaded": os.path.exists(MODEL_PATH)})

@app.route('/chat', methods=['POST'])
def chat():
    data = request.get_json()
    messages = data.get("messages", [])
    if not messages:
        return jsonify({"error": "Missing 'messages' array"}), 400
    
    # Build prompt from conversation history
    prompt = ""
    for msg in messages:
        role = msg.get("role", "user")
        content = msg.get("content", "")
        prompt += f"[{role.upper()}] {content}\n"
    prompt += "[ASSISTANT]"
    
    max_tokens = data.get("max_tokens", 256)
    temperature = data.get("temperature", 0.7)
    
    response = llm(
        prompt,
        max_tokens=max_tokens,
        temperature=temperature,
        stop=["[USER]", "[ASSISTANT]"],
        echo=False
    )
    
    reply_text = response["choices"][0]["text"].strip()
    usage = response.get("usage", {})
    
    return jsonify({
        "reply": reply_text,
        "tokens_used": usage.get("completion_tokens", 0),
        "finish_reason": response["choices"][0].get("finish_reason", "unknown")
    })

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8003)

chatbot/Dockerfile:

FROM python:3.11-slim

WORKDIR /app

RUN apt-get update && apt-get install -y wget && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Create models directory and download a quantized LLaMA 2 7B model
RUN mkdir -p /app/models && \
    wget -O /app/models/llama-2-7b-chat.Q4_K_M.gguf \
    https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf

COPY app.py .

EXPOSE 8003

ENV MODEL_PATH=/app/models/llama-2-7b-chat.Q4_K_M.gguf

CMD ["gunicorn", "--bind", "0.0.0.0:8003", "--workers", "1", "--timeout", "180", "app:app"]

The model download at build time creates a large Docker image (~5 GB), but eliminates cold-start delays. For production, consider mounting models via volumes instead (covered in best practices).

4. Setting Up the NGINX Reverse Proxy

The reverse proxy gives us a single entry point and routes requests to the appropriate agent based on URL path.

nginx/default.conf:

server {
    listen 80;
    server_name _;

    # Summarizer agent
    location /summarizer/ {
        proxy_pass http://summarizer:8001/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 120s;
    }

    # Image generator agent
    location /imagegen/ {
        proxy_pass http://imagegen:8002/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
    }

    # Chatbot agent
    location /chatbot/ {
        proxy_pass http://chatbot:8003/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 180s;
    }

    # Health check endpoint aggregated
    location /health {
        return 200 "OK - Multi-Agent Platform\n";
        add_header Content-Type text/plain;
    }
}

nginx/Dockerfile:

FROM nginx:1.25-alpine
COPY default.conf /etc/nginx/conf.d/default.conf

5. Writing the Docker Compose Configuration

This is the central file that ties all agents together with networking, GPU allocation, restart policies, and resource constraints.

docker-compose.yml:

version: '3.8'

services:
  # ---- REVERSE PROXY ----
  nginx:
    build: ./nginx
    ports:
      - "80:80"
    networks:
      - agent_net
    depends_on:
      summarizer:
        condition: service_started
      imagegen:
        condition: service_started
      chatbot:
        condition: service_started
    restart: unless-stopped

  # ---- SUMMARIZER AGENT (CPU) ----
  summarizer:
    build: ./summarizer
    expose:
      - "8001"
    networks:
      - agent_net
    environment:
      - TRANSFORMERS_CACHE=/app/.cache
      - HF_HOME=/app/.cache
    volumes:
      - summarizer_cache:/app/.cache
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          cpus: '1.0'
          memory: 1G

  # ---- IMAGE GENERATOR AGENT (GPU) ----
  imagegen:
    build: ./imagegen
    expose:
      - "8002"
    networks:
      - agent_net
    environment:
      - TRANSFORMERS_CACHE=/app/.cache
      - HF_HOME=/app/.cache
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - imagegen_cache:/app/.cache
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G
        reservations:
          cpus: '2.0'
          memory: 4G
        # GPU reservation via Docker Compose (requires Compose v2.3+ with GPU support)
    # GPU access declaration
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=0

  # ---- CHATBOT AGENT (CPU, large model) ----
  chatbot:
    build: ./chatbot
    expose:
      - "8003"
    networks:
      - agent_net
    environment:
      - MODEL_PATH=/app/models/llama-2-7b-chat.Q4_K_M.gguf
    volumes:
      - chatbot_models:/app/models
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 6G
        reservations:
          cpus: '2.0'
          memory: 3G

volumes:
  summarizer_cache:
  imagegen_cache:
  chatbot_models:

networks:
  agent_net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16

Key details in this Compose file:

• Resource limits: Each agent has CPU and memory boundaries to prevent a runaway process from starving others.
• GPU declaration: The imagegen service uses runtime: nvidia and NVIDIA_VISIBLE_DEVICES=0 to access GPU 0 exclusively. If you have multiple GPUs, you could assign different agents to different GPU indices.
• Named volumes: Model caches and downloaded weights persist across container rebuilds, saving bandwidth and startup time.
• Internal network: The agent_net bridge isolates inter-service traffic from the host's network namespace.

6. Building and Deploying the Stack

With all files in place, navigate to the multi-agent-server/ directory and run:

# Build all images (this will take 10-20 minutes depending on model downloads)
docker compose build

# Start the entire stack in detached mode
docker compose up -d

# Check that all services are running
docker compose ps

# View logs for all agents
docker compose logs -f

# Tail logs for a specific agent
docker compose logs -f summarizer

Once running, test each agent through the reverse proxy:

# Test summarizer
curl -X POST http://localhost/summarizer/summarize \
  -H "Content-Type: application/json" \
  -d '{"text": "Artificial intelligence is transforming industries across the globe. From healthcare to finance, AI-powered systems are making predictions, automating tasks, and uncovering insights at unprecedented scales. However, deploying multiple AI models in production requires careful orchestration to manage resource usage and ensure reliability.", "max_length": 60}'

# Test image generator
curl -X POST http://localhost/imagegen/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "A serene mountain lake at sunset", "steps": 20}'

# Test chatbot
curl -X POST http://localhost/chatbot/chat \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is Docker?"}], "max_tokens": 100}'

Advanced Configuration: GPU Sharing Strategies

When multiple agents need GPU access, you have several options beyond the simple single-GPU assignment shown above:

Option A: Multi-GPU Server Partitioning

On a server with multiple GPUs (e.g., 4× NVIDIA A10), assign specific GPUs to each agent:

# In docker-compose.yml for each GPU-dependent service:
services:
  imagegen:
    environment:
      - NVIDIA_VISIBLE_DEVICES=0
    runtime: nvidia
    # ... rest of config
  
  transcriber:   # another hypothetical agent
    environment:
      - NVIDIA_VISIBLE_DEVICES=1
    runtime: nvidia
    # ... rest of config

Option B: Time-Sliced GPU Sharing (Multi-Process Service)

For scenarios where multiple agents must share a single GPU, use NVIDIA's MPS (Multi-Process Service) or configure GPU memory limits via environment variables:

# In docker-compose.yml
services:
  imagegen:
    environment:
      - NVIDIA_VISIBLE_DEVICES=0
      - CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
      - CUDA_MPS_LOG_DIRECTORY=/var/log/nvidia-mps
    runtime: nvidia
    volumes:
      - /tmp/nvidia-mps:/tmp/nvidia-mps
    deploy:
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
                kind: 'NVIDIA_GPU'
                value: 0.5   # Request 50% of GPU resources

For production, NVIDIA's GPU Operator on Kubernetes provides finer-grained GPU slicing, but for Docker Compose, the NVIDIA_VISIBLE_DEVICES environment variable with MPS offers a pragmatic middle ground.

Best Practices for Production Deployments

1. Use Pre-Built Images with CI/CD

Don't build images that download 5 GB models on every deployment. Instead, push pre-built images to a container registry (AWS ECR, Docker Hub, GHCR) via a CI pipeline:

# Example CI step (GitHub Actions snippet)
- name: Build and push chatbot image
  run: |
    docker build -t ghcr.io/myorg/chatbot-agent:latest ./chatbot
    docker push ghcr.io/myorg/chatbot-agent:latest

Then reference the pre-built image in Compose:

chatbot:
  image: ghcr.io/myorg/chatbot-agent:latest
  # build: ./chatbot  # Comment out or remove the build section

2. Externalize Model Storage with Volumes

For large models (>2 GB), avoid baking them into the Docker image. Instead, mount them as read-only volumes or use a dedicated model cache volume:

chatbot:
  image: ghcr.io/myorg/chatbot-agent:latest
  volumes:
    - /data/models/llama-2-7b-chat.Q4_K_M.gguf:/app/models/llama-2-7b-chat.Q4_K_M.gguf:ro
    - chatbot_cache:/app/.cache

This keeps images small (under 500 MB) and allows model updates without rebuilding containers.

3. Implement Health Checks and Auto-Restart

Add Docker health checks so Compose can restart unhealthy containers automatically:

services:
  summarizer:
    # ... other config
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped

4. Centralize Logging and Monitoring

For a single-server multi-agent setup, use Docker's built-in logging drivers to ship logs to a central location or use a monitoring sidecar:

# In docker-compose.yml under each service:
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
    labels: "agent-name"

For more advanced observability, integrate with Prometheus exporters or add a Grafana Loki sidecar container.

5. Secure Inter-Agent Communication

While the agent_net bridge network isolates traffic, consider adding mutual TLS for sensitive agent-to-agent calls in production. At minimum, restrict which agents can reach external networks:

services:
  summarizer:
    networks:
      agent_net:
        # No external network access needed
    # Prevent internet access for this container
    # (requires custom network configuration with internal: true)

6. Right-Size Resource Limits

Monitor actual usage with docker stats over 24 hours, then adjust Compose resource limits to reflect real consumption with a 20% buffer:

# Check resource usage of running containers
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

Over-provisioning memory limits wastes resources; under-provisioning triggers OOM kills. Iterate based on observed metrics.

Conclusion

Hosting multiple AI agents on a single server with Docker transforms a potentially chaotic collection of scripts and models into a well-orchestrated, resource-efficient microservices platform. Through Docker Compose, you define each agent's runtime environment, resource boundaries, GPU access, and networking in a declarative configuration file that can be version-controlled and reproduced anywhere.

In this tutorial, you built a complete multi-agent stack—a summarizer, an image generator, and a chatbot—each running in its own isolated container, communicating through an NGINX reverse proxy, and sharing a single server's CPU and GPU resources. You learned how to pre-cache models at build time, configure GPU passthrough, set resource limits, and structure your project for maintainability.

The patterns described here scale naturally: add new agents by creating additional services in the Compose file, scale horizontally by placing the Compose stack behind a load balancer on multiple servers, or graduate to Kubernetes when you need advanced orchestration features like auto-scaling and rolling updates. The core principle remains the same—each AI agent is a self-contained Docker service that can be developed, tested, and deployed independently while coexisting harmoniously on shared infrastructure.