Introduction to Multi-Modal Models
Multi-modal models are AI systems capable of processing and reasoning over more than one type of input — typically text and images, but increasingly audio, video, and structured data as well. Unlike traditional language models that only accept text tokens, multi-modal models can look at a photo, read a chart, or analyze a screenshot and answer questions about it in natural language.
Popular open-weight multi-modal models include Meta's LLaVA (Large Language and Vision Assistant), Qwen-VL, CogVLM, Mistral's Pixtral, and Llama 3.2 Vision. These models combine a vision encoder (often CLIP or SigLIP) with a large language model, bridging the two with a projection layer that translates visual features into the language model's embedding space.
Why Run Multi-Modal Models Locally?
Cloud APIs from OpenAI, Google, and Anthropic make multi-modal inference easy, but running models on your own hardware offers several compelling advantages:
- Privacy: Sensitive images, medical scans, internal documents, and proprietary screenshots never leave your machine.
- Cost: No per-token or per-image fees. You pay once for hardware and run unlimited inferences.
- Latency: No network round-trips. Local inference can be faster for interactive applications.
- Offline capability: Works in air-gapped environments, on edge devices, or in field deployments.
- Customization: Full control over quantization, system prompts, fine-tuning, and inference parameters.
- No vendor lock-in: You own the entire pipeline and are not subject to API deprecations or policy changes.
Prerequisites and Hardware Considerations
Multi-modal models are heavier than text-only models because they include a vision encoder in addition to the language model. Before you begin, assess your hardware:
- Minimum: 8 GB VRAM GPU (or 16 GB unified memory on Apple Silicon) for 7B parameter quantized models like LLaVA 1.5 7B.
- Recommended: 16–24 GB VRAM for comfortable inference of 8B–13B vision models with high-resolution images.
- CPU-only: Possible with llama.cpp using GGUF quantized models, but expect significantly slower performance.
- Apple Silicon: Macs with M-series chips and 16+ GB unified memory are excellent for local multi-modal inference via Ollama or MLX.
Install the core tools you will need. The examples below use Python 3.10+ along with Ollama and Hugging Face libraries.
# Install Ollama (macOS / Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Install Python dependencies
pip install ollama transformers torch accelerate pillow requests
Option 1: Running LLaVA with Ollama
Ollama is the simplest way to run multi-modal models locally. It handles model downloading, quantization, and serving through a clean CLI and REST API. LLaVA is one of the best-supported vision models in the Ollama ecosystem.
Step 1: Pull the Model
# Pull the 7B LLaVA model (approximately 4.7 GB download)
ollama pull llava
# Or pull the smaller, faster llava-llama3 variant
ollama pull llava-llama3
# List installed models
ollama list
Step 2: Chat from the Command Line
# Start an interactive chat and attach an image
ollama run llava "Describe what you see in this image: /path/to/photo.jpg"
Step 3: Use the Python Client
For application development, use the official Ollama Python client. This script sends an image and a prompt to the locally running model.
import ollama
import base64
from pathlib import Path
def encode_image(image_path: str) -> str:
"""Read an image file and return its base64-encoded string."""
with open(image_path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
image_path = "sample.jpg"
image_b64 = encode_image(image_path)
response = ollama.chat(
model="llava",
messages=[
{
"role": "user",
"content": "What objects are visible in this image? List them as bullet points.",
"images": [image_b64],
}
],
)
print(response["message"]["content"])
Step 4: Use the REST API Directly
Ollama exposes a local HTTP API on port 11434, which you can call from any language.
curl http://localhost:11434/api/chat -d '{
"model": "llava",
"messages": [
{
"role": "user",
"content": "Is there a cat in this image?",
"images": ["'$(base64 -i cat.jpg)'"]
}
],
"stream": false
}'
Option 2: Running Models with Hugging Face Transformers
For maximum flexibility — including fine-tuning, custom pipelines, and access to the latest model architectures — use Hugging Face Transformers. This approach gives you direct control over tokenization, image preprocessing, and generation parameters.
Running Qwen2-VL
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from PIL import Image
import torch
# Load model in half precision to save VRAM
model_id = "Qwen/Qwen2-VL-7B-Instruct"
model = Qwen2VLForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)
# Prepare the image and prompt
image = Image.open("chart.png").convert("RGB")
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "Extract all the data points from this chart and format them as a table."},
],
}
]
# Apply the chat template and process inputs
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(
text=[text],
images=[image],
padding=True,
return_tensors="pt",
).to("cuda")
# Generate the response
generated_ids = model.generate(**inputs, max_new_tokens=512)
output_text = processor.batch_decode(
generated_ids[:, inputs["input_ids"].shape[1]:],
skip_special_tokens=True,
)
print(output_text[0])
Running Llama 3.2 Vision
from transformers import MllamaForConditionalGeneration, AutoProcessor
from PIL import Image
import torch
model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct"
model = MllamaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)
image = Image.open("document.png").convert("RGB")
messages = [
{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": "Summarize the key information in this document."},
]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0], skip_special_tokens=True))
Option 3: Running GGUF Models with llama.cpp
llama.cpp provides efficient CPU and GPU inference using GGUF quantized models. It is ideal for resource-constrained environments and supports multi-modal models through its mmproj (multimodal projector) system.
Build and Run
# Clone and build llama.cpp with CUDA support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_CUDA=1
# Download a quantized LLaVA model and its vision projector
# (from a Hugging Face GGUF mirror such as jartine/llava-v1.5-7B-GGUF)
# You need two files: the main model .gguf and the mmproj .gguf
# Run inference with an image
./llama-cli \
-m llava-v1.5-7b-Q4_K_M.gguf \
--mmproj mmproj-model-f16.gguf \
--image sample.jpg \
-p "Describe this image in detail." \
-n 256
You can also run llama.cpp as a server with an OpenAI-compatible API, which lets you swap local inference into applications originally built for cloud APIs.
./llama-server \
-m llava-v1.5-7b-Q4_K_M.gguf \
--mmproj mmproj-model-f16.gguf \
--port 8080
# Then call it like the OpenAI API:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llava",
"messages": [{"role": "user", "content": "What is in this image?"}],
"image_data": [{"data": "'$(base64 -w0 sample.jpg)'"}]
}'
Building a Practical Multi-Modal Application
Let's build a small command-line utility that analyzes any image file and answers a user question about it. This example uses Ollama as the backend and includes error handling and image validation.
import ollama
import base64
import sys
from pathlib import Path
from PIL import Image
SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif"}
def validate_image(image_path: str) -> bool:
"""Check that the file exists, is a supported format, and can be opened."""
path = Path(image_path)
if not path.exists():
raise FileNotFoundError(f"Image not found: {image_path}")
if path.suffix.lower() not in SUPPORTED_FORMATS:
raise ValueError(f"Unsupported format: {path.suffix}")
try:
with Image.open(path) as img:
img.verify()
except Exception as e:
raise ValueError(f"Invalid image file: {e}")
return True
def analyze_image(image_path: str, question: str, model: str = "llava") -> str:
"""Send an image and question to a local multi-modal model."""
validate_image(image_path)
with open(image_path, "rb") as f:
image_b64 = base64.b64encode(f.read()).decode("utf-8")
response = ollama.chat(
model=model,
messages=[
{
"role": "system",
"content": "You are a helpful visual assistant. Answer concisely and accurately.",
},
{
"role": "user",
"content": question,
"images": [image_b64],
},
],
options={"temperature": 0.3, "num_ctx": 4096},
)
return response["message"]["content"]
if __name__ == "__main__":
if len(sys.argv) < 3:
print("Usage: python analyze.py <image_path> <question>")
sys.exit(1)
image_path = sys.argv[1]
question = sys.argv[2]
try:
answer = analyze_image(image_path, question)
print(f"\nQuestion: {question}")
print(f"Answer: {answer}\n")
except Exception as e:
print(f"Error: {e}")
sys.exit(1)
Run it from the terminal:
python analyze.py screenshot.png "What error is shown in this terminal output?"
Best Practices for Local Multi-Modal Inference
Choose the Right Quantization Level
Quantization reduces model size and memory usage at the cost of some accuracy. For most use cases, Q4_K_M or Q5_K_M quantization offers an excellent balance. Use Q8 or unquantized FP16 only when you need maximum accuracy and have VRAM to spare.
Resize Images Before Sending
Multi-modal models process images in fixed tile sizes. Sending a 12-megapixel photo wastes memory and compute. Resize images to a reasonable resolution (typically 1024×1024 or smaller) before inference, unless fine detail is critical to your task.
from PIL import Image
def resize_for_inference(image_path: str, max_size: int = 1024) -> Image.Image:
img = Image.open(image_path).convert("RGB")
ratio = max_size / max(img.size)
if ratio < 1:
new_size = (int(img.size[0] * ratio), int(img.size[1] * ratio))
img = img.resize(new_size, Image.LANCZOS)
return img
Manage Context Length
Images consume a large number of tokens in the context window — often 500 to 2000 tokens per image depending on the model and resolution. When processing multiple images or long conversations, monitor your context usage and truncate history to avoid errors and slowdowns.
Use Streaming for Interactive Applications
For chat interfaces and real-time tools, stream tokens as they are generated. This dramatically improves perceived performance.
import ollama
stream = ollama.chat(
model="llava",
messages=[{"role": "user", "content": "Describe this image.", "images": [image_b64]}],
stream=True,
)
for chunk in stream:
print(chunk["message"]["content"], end="", flush=True)
Cache Models and Preprocess Once
If you are running a Hugging Face model in a long-running application, load the model and processor once at startup and reuse them. Repeatedly calling from_pretrained will reload weights from disk every time, adding seconds of latency to each request.
Benchmark Before Deploying
Always measure tokens-per-second and image processing latency on your target hardware before committing to a model. A model that runs well on a developer workstation may be unusably slow on a production edge device. Tools like ollama benchmark and simple Python timers help you make data-driven decisions.
Handle Model-Specific Quirks
Different multi-modal models have different strengths. LLaVA is good at general image description, Qwen2-VL excels at document and chart understanding, and Llama 3.2 Vision is strong at reasoning. Test multiple models on your specific task and pick the one that performs best, not just the one with the most parameters.
Conclusion
Running multi-modal models locally has never been more accessible. With tools like Ollama, Hugging Face Transformers, and llama.cpp, you can deploy powerful vision-language models on consumer hardware in minutes. Whether you need privacy for sensitive data, offline capability for edge deployments, or simply want to avoid recurring API costs, the open-source ecosystem now provides production-viable options. Start with Ollama and LLaVA for quick prototyping, move to Transformers when you need fine-grained control or fine-tuning, and use llama.cpp for maximum efficiency on constrained hardware. By following the best practices around quantization, image preprocessing, and context management, you can build robust multi-modal applications that run entirely on your own machine.