← Back to DevBytes
Docker August 19, 2026 10 min read

How to Deploy Local LLMs with Ollama and Docker

Introduction to Local LLM Deployment with Ollama and Docker

Large Language Models (LLMs) have transformed how developers build intelligent applications, but relying on cloud APIs introduces latency, cost, and privacy concerns. Running LLMs locally solves these problems, and Ollama has emerged as one of the most popular tools for simplifying local LLM deployment. When combined with Docker, you get a reproducible, portable, and production-ready environment for serving language models on your own infrastructure.

In this tutorial, you will learn how to deploy local LLMs using Ollama and Docker, from a basic single-container setup to a multi-container architecture with a custom model configuration. By the end, you will have a working local LLM service accessible via a REST API.

What Is Ollama?

Ollama is an open-source tool that lets you run large language models locally on your machine. It abstracts away the complexity of model quantization, weight downloading, and inference engine configuration. With a single command, you can pull and run models like Llama 3, Mistral, Phi-3, Gemma, and many others.

Ollama provides:

Why It Matters

Deploying LLMs locally with Ollama and Docker matters for several key reasons:

Privacy and Data Sovereignty

When you send prompts to a cloud-based LLM API, your data leaves your infrastructure. For industries handling sensitive information — healthcare, finance, legal, or internal enterprise data — local deployment ensures that no prompt or response ever leaves your network.

Cost Control

Cloud LLM APIs charge per token. For high-volume applications, costs can spiral quickly. A local deployment has a fixed hardware cost and zero per-query fees, making it economically viable for bulk processing tasks like document summarization, log analysis, or code review.

Latency Reduction

Local inference eliminates network round trips. For interactive applications where response time is critical, running the model on-premises can dramatically reduce latency, especially when the model is warm and loaded in memory.

Reproducibility with Docker

Docker containers encapsulate the entire runtime environment. This means your Ollama setup behaves identically whether it runs on a developer laptop, a CI server, or a production GPU machine. Docker also makes it easy to scale horizontally and integrate Ollama into existing containerized architectures.

Prerequisites

Before you begin, ensure you have the following:

Verify your Docker installation:

docker --version
docker compose version

Running Ollama in Docker: The Quick Start

The fastest way to get Ollama running is with a single Docker command. Ollama publishes an official image on Docker Hub.

docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

Let's break down this command:

Check that the container is running:

docker ps

You should see the Ollama container listed with a status of "Up." Test the API endpoint:

curl http://localhost:11434

If everything is working, you will receive the response Ollama is running.

Pulling and Running Your First Model

Once Ollama is running, you can pull a model using the Ollama CLI inside the container. Let's start with Llama 3.2, a lightweight and capable model:

docker exec -it ollama ollama pull llama3.2

This downloads the quantized model weights. The download size depends on the model — Llama 3.2 (3B parameters) is approximately 2 GB. Once the download completes, run the model interactively:

docker exec -it ollama ollama run llama3.2

You can now type prompts directly in the terminal. To exit the interactive session, type /bye or press Ctrl+D.

Using the REST API

Ollama exposes a REST API on port 11434. Here is how to generate a completion using curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what a Docker container is in one sentence.",
  "stream": false
}'

For chat-style interactions with message history, use the /api/chat endpoint:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python function to reverse a string."}
  ],
  "stream": false
}'

Setting "stream": false returns the complete response as a single JSON object. If you omit this parameter or set it to true, Ollama streams the response token by token via Server-Sent Events, which is useful for real-time UI updates.

Using Docker Compose for a Managed Setup

For any real-world deployment, Docker Compose provides better manageability than raw docker run commands. Create a docker-compose.yml file:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped
    # Uncomment the following lines for GPU support
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  ollama_data:

Start the service:

docker compose up -d

View logs to confirm a healthy startup:

docker compose logs -f ollama

Stop the service when you are done:

docker compose down

The named volume ollama_data persists across container restarts, so your downloaded models remain available even after you recreate the container.

GPU Acceleration with Docker

CPU inference works but is slow for larger models. If you have an NVIDIA GPU, you can enable GPU acceleration. First, install the NVIDIA Container Toolkit on your host:

# Ubuntu/Debian
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then use the GPU-enabled Compose configuration by uncommenting the deploy section shown earlier. Verify GPU access inside the container:

docker exec -it ollama nvidia-smi

If the NVIDIA driver and toolkit are correctly configured, you will see the GPU information table. GPU inference can be 10x to 50x faster than CPU inference depending on the model size.

Creating a Custom Model with a Modelfile

Ollama supports custom model configurations through a Modelfile, which is similar to a Dockerfile. You can define a system prompt, adjust parameters like temperature and context window, and even import custom weights.

Create a file named Modelfile:

FROM llama3.2

# Set a system prompt that defines the model's behavior
SYSTEM """
You are an expert software engineer. You provide concise, accurate answers
with working code examples. Always explain your reasoning briefly before
showing code.
"""

# Adjust generation parameters
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER stop "</answer>"

# Add a custom template for structured output
TEMPLATE """
{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>
"""

Build the custom model inside the running Ollama container. First, copy the Modelfile into the container:

docker cp Modelfile ollama:/tmp/Modelfile
docker exec -it ollama ollama create my-coder -f /tmp/Modelfile

Now you can run your custom model:

curl http://localhost:11434/api/chat -d '{
  "model": "my-coder",
  "messages": [
    {"role": "user", "content": "Write a Dockerfile for a Node.js app."}
  ],
  "stream": false
}'

The custom model inherits the base model's capabilities but applies your system prompt and parameter tuning to every interaction.

Integrating Ollama with Other Containers

In a typical architecture, Ollama serves as a backend inference engine for a frontend application or API gateway. Here is a Compose file that runs Ollama alongside a simple Python API wrapper:

version: "3.9"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    restart: unless-stopped
    ports:
      - "11434:11434"

  api:
    build: ./api
    container_name: llm-api
    environment:
      - OLLAMA_URL=http://ollama:11434
      - MODEL_NAME=llama3.2
    ports:
      - "8000:8000"
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:

Create the API service in ./api/main.py using FastAPI:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import httpx
import os

app = FastAPI()

OLLAMA_URL = os.getenv("OLLAMA_URL", "http://ollama:11434")
MODEL_NAME = os.getenv("MODEL_NAME", "llama3.2")

class ChatRequest(BaseModel):
    message: str

class ChatResponse(BaseModel):
    response: str

@app.post("/chat", response_model=ChatResponse)
async def chat(req: ChatRequest):
    try:
        async with httpx.AsyncClient(timeout=120.0) as client:
            result = await client.post(
                f"{OLLAMA_URL}/api/chat",
                json={
                    "model": MODEL_NAME,
                    "messages": [{"role": "user", "content": req.message}],
                    "stream": False,
                },
            )
            result.raise_for_status()
            data = result.json()
            return ChatResponse(response=data["message"]["content"])
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health():
    return {"status": "ok"}

Create ./api/requirements.txt:

fastapi==0.115.0
uvicorn==0.30.6
httpx==0.27.2

Create ./api/Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Start the full stack:

docker compose up -d --build

Before using the API, pull the model into the Ollama container:

docker exec -it ollama ollama pull llama3.2

Test the wrapper API:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What are the benefits of containerization?"}'

The API container communicates with Ollama over the internal Docker network using the service name ollama as the hostname. This is a clean, decoupled architecture where the inference engine and application logic are independently scalable.

Preloading Models on Startup

In production, you want models to be available immediately when the container starts, without manual ollama pull commands. You can create a custom Ollama image that preloads models:

FROM ollama/ollama:latest

# Copy a startup script
COPY start.sh /start.sh
RUN chmod +x /start.sh

CMD ["/start.sh"]

Create start.sh:

#!/bin/bash
set -e

# Start Ollama in the background
ollama serve &
OLLAMA_PID=$!

# Wait for the API to be ready
echo "Waiting for Ollama to start..."
until curl -s http://localhost:11434 >/dev/null 2>&1; do
  sleep 1
done

echo "Ollama is ready. Pulling models..."

# Preload models
ollama pull llama3.2
ollama pull nomic-embed-text

echo "Models loaded successfully."

# Wait for the background process
wait $OLLAMA_PID

Build and run:

docker build -t ollama-preloaded .
docker run -d --name ollama -p 11434:11434 -v ollama_data:/root/.ollama ollama-preloaded

This approach ensures that every time the container starts, the required models are available without manual intervention.

Best Practices

Use Persistent Volumes

Always mount a volume at /root/.ollama. Without persistence, every container recreation triggers a full model re-download, which wastes bandwidth and time. Named volumes are preferred over bind mounts for model storage because Docker manages them and they avoid permission issues.

Set Resource Limits

LLM inference is resource-intensive. In a shared environment, set memory and CPU limits to prevent Ollama from starving other services:

services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        limits:
          memory: 16G
          cpus: "4"

Secure the API Endpoint

By default, Ollama listens on all interfaces with no authentication. In production, either bind it to localhost only (127.0.0.1:11434) or place it behind a reverse proxy with authentication. Here is a simple nginx configuration snippet:

server {
    listen 8080;

    location / {
        proxy_pass http://ollama:11434;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;

        # Optional: add basic auth
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}

Choose the Right Model Size

Selecting an appropriately sized model is critical. A model that is too large will cause out-of-memory errors or extreme latency. A model that is too small may produce low-quality output. As a general guideline:

Monitor Container Health

Add a health check to your Compose file so Docker can detect and restart unhealthy containers:

services:
  ollama:
    image: ollama/ollama:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    restart: unless-stopped

Use Streaming for Better UX

For user-facing applications, enable streaming responses. Ollama streams tokens as they are generated, which gives users immediate feedback instead of waiting for the entire response. Most HTTP clients support streaming via Server-Sent Events or chunked transfer encoding.

Keep Images Updated

Ollama releases updates frequently with performance improvements and new model support. Periodically pull the latest image and recreate your containers:

docker compose pull
docker compose up -d

Always test updates in a staging environment first, as new versions may change API behavior or model compatibility.

Conclusion

Deploying local LLMs with Ollama and Docker gives you a powerful, private, and cost-effective alternative to cloud-based AI APIs. Ollama handles the complexity of model management and inference, while Docker provides the portability, reproducibility, and orchestration capabilities needed for production deployments. By following the patterns in this tutorial — persistent volumes, Docker Compose orchestration, custom Modelfiles, GPU acceleration, and proper security practices — you can build a robust local LLM infrastructure that scales from a developer laptop to a production server. Whether you are building a coding assistant, a document analysis pipeline, or a chatbot for internal use, this stack provides a solid foundation that keeps your data private, your costs predictable, and your architecture clean.

— Ad —

Google AdSense will appear here after approval

← Back to all articles