How to Implement Multi-Modal RAG with CLIP Embeddings

How to Implement Multi-Modal RAG with CLIP Embeddings

Retrieval-Augmented Generation (RAG) has become the standard pattern for grounding large language models in private or up-to-date data. Traditional RAG pipelines, however, are text-only: they chunk documents, embed the chunks, store them in a vector database, and retrieve relevant passages at query time. But real-world knowledge is rarely text-only — it lives in PDFs with charts, product catalogs with images, medical records with scans, and slide decks with diagrams. Multi-Modal RAG extends the retrieval step to handle both images and text in a single unified embedding space, allowing the system to retrieve the most relevant content regardless of its modality.

The key enabler for this unified retrieval is CLIP (Contrastive Language-Image Pre-training), a model released by OpenAI that maps images and text into the same vector space. When you embed a photo of a golden retriever and the sentence "a fluffy golden dog playing fetch," their vectors sit close together. This shared space is what makes multi-modal retrieval possible: you can search an image library with a text query, search a text corpus with an image query, or mix both in a single index.

Why Multi-Modal RAG Matters

Text-only RAG breaks down in several common scenarios. If a user asks "which product has the red packaging with a leaf logo," a text index cannot match the visual description to product photos. If a support question references an error dialog screenshot, the answer may live in a troubleshooting diagram that has no text equivalent. Multi-Modal RAG solves these cases by letting the retriever pull from any modality and letting the generator reason over mixed inputs.

• Cross-modal search: Find images using natural language queries, or find text passages using reference images.
• Richer grounding: The LLM receives both the relevant paragraph and the relevant figure, producing more accurate answers.
• Simpler pipelines: One index, one embedding model, one retrieval call — no need to maintain separate text and image stores.
• Better user experience: Users can upload a photo and ask a question about it, and the system retrieves supporting context from both text and image corpora.

Architecture Overview

A Multi-Modal RAG system built on CLIP has four main components. First, an ingestion pipeline extracts text chunks and images from source documents, embeds both with CLIP, and stores the vectors alongside metadata in a vector database. Second, a retrieval pipeline embeds the user query (text or image) with the same CLIP model and performs a similarity search. Third, a context assembly step gathers the top-k results, loads the original payloads, and formats them for the generator. Fourth, a generation pipeline sends the assembled context — including images, if the LLM supports vision — to a model that produces the final answer.

The diagram below summarizes the flow:

Source docs ──▶ Extract text + images
                    │
                    ▼
              CLIP embed (text + image)
                    │
                    ▼
            Vector DB (unified index)
                    │
   User query ──────┤
   (text/image)     ▼
              CLIP embed query
                    │
                    ▼
            Similarity search ──▶ top-k chunks/images
                    │
                    ▼
        Assemble context (text + image refs)
                    │
                    ▼
            Vision LLM ──▶ Answer

Prerequisites and Setup

You will need Python 3.9 or later and a handful of libraries. The example below uses open_clip_torch for the CLIP model (an open-source implementation that supports multiple pretrained variants), chromadb for the vector store, Pillow for image handling, and openai for the final generation step. Install them with:

pip install open_clip_torch chromadb Pillow openai torch torchvision

You will also need an OpenAI API key if you want to use GPT-4o for the generation step. Set it as an environment variable:

export OPENAI_API_KEY="sk-your-key-here"

Step 1: Loading the CLIP Model

The first step is to load a CLIP model and its preprocessing transform. open_clip makes this straightforward. We will use the ViT-B-32 variant with OpenAI weights — it offers a good balance of speed and quality, and its 512-dimensional embeddings are easy to store. For production workloads with higher accuracy requirements, consider ViT-L-14 or ViT-H-14.

import open_clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32",
    pretrained="openai"
)
model = model.to(device)
model.eval()

tokenizer = open_clip.get_tokenizer("ViT-B-32")

def embed_text(text: str) -> list[float]:
    with torch.no_grad():
        tokens = tokenizer([text]).to(device)
        vec = model.encode_text(tokens)
        vec = vec / vec.norm(dim=-1, keepdim=True)
    return vec[0].cpu().tolist()

def embed_image(image_path: str) -> list[float]:
    img = preprocess(Image.open(image_path).convert("RGB")).unsqueeze(0).to(device)
    with torch.no_grad():
        vec = model.encode_image(img)
        vec = vec / vec.norm(dim=-1, keepdim=True)
    return vec[0].cpu().tolist()

Notice that we L2-normalize every embedding. This is important because CLIP was trained with cosine similarity, and most vector databases compute dot-product or Euclidean distance. Normalizing the vectors makes dot-product equivalent to cosine similarity, which gives the most accurate retrieval results.

Step 2: Building the Ingestion Pipeline

The ingestion pipeline extracts text chunks and images from your source material, embeds each one, and writes them to the vector database. For this tutorial we will keep extraction simple: text comes in as a list of paragraphs, and images come in as a list of file paths. In a real system you would parse PDFs, slide decks, or HTML using libraries like pdfplumber, python-pptx, or beautifulsoup.

import chromadb

client = chromadb.PersistentClient(path="./mm_rag_db")
collection = client.get_or_create_collection(
    name="multimodal_docs",
    metadata={"hnsw:space": "cosine"}
)

def ingest_text_chunks(chunks: list[str], source: str) -> None:
    for i, chunk in enumerate(chunks):
        vec = embed_text(chunk)
        collection.add(
            ids=[f"{source}_text_{i}"],
            embeddings=[vec],
            documents=[chunk],
            metadatas=[{"modality": "text", "source": source, "index": i}]
        )

def ingest_images(image_paths: list[str], source: str) -> None:
    for i, path in enumerate(image_paths):
        vec = embed_image(path)
        collection.add(
            ids=[f"{source}_img_{i}"],
            embeddings=[vec],
            documents=[path],  # store path so we can reload the image later
            metadatas=[{"modality": "image", "source": source, "index": i, "path": path}]
        )

Each record stores the embedding, a document payload, and metadata. The modality field lets us filter or route results at retrieval time, and the path field on image records lets us reload the original file for the vision LLM. Putting both modalities in the same collection is what enables cross-modal search — a text query can surface image results and vice versa.

Now ingest some sample data:

text_chunks = [
    "The Model X has a panoramic windshield that extends over the front seats.",
    "Our return policy allows refunds within 30 days of purchase with a receipt.",
    "The circuit board uses a 12-volt power supply and a heat sink for cooling."
]

image_paths = [
    "./data/model_x_interior.jpg",
    "./data/return_policy_poster.png",
    "./data/circuit_board_diagram.jpg"
]

ingest_text_chunks(text_chunks, source="knowledge_base")
ingest_images(image_paths, source="knowledge_base")
print(f"Total records in collection: {collection.count()}")

Step 3: Implementing Multi-Modal Retrieval

Retrieval works the same way regardless of whether the query is text or an image — you embed it with CLIP and run a similarity search. The only difference is which embedding function you call. This symmetry is the core benefit of a shared embedding space.

def retrieve(query: str | None = None, image_path: str | None = None, top_k: int = 5):
    if query is not None:
        query_vec = embed_text(query)
    elif image_path is not None:
        query_vec = embed_image(image_path)
    else:
        raise ValueError("Provide either a text query or an image path.")

    results = collection.query(
        query_embeddings=[query_vec],
        n_results=top_k,
        include=["documents", "metadatas", "distances"]
    )

    retrieved = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        results["distances"][0]
    ):
        retrieved.append({
            "payload": doc,
            "modality": meta["modality"],
            "source": meta["source"],
            "distance": dist,
            "image_path": meta.get("path")
        })
    return retrieved

Test it with a text query that should match both a text chunk and an image:

hits = retrieve(query="car with a big glass roof", top_k=3)
for hit in hits:
    print(f"[{hit['modality']}] dist={hit['distance']:.4f} -> {hit['payload'][:80]}")

You should see the Model X text chunk and the interior photo near the top of the results, even though the query wording differs from both. That is CLIP's semantic alignment at work.

Step 4: Assembling Context for the Generator

The retrieved items are raw embeddings results — they need to be turned into something a language model can consume. For text hits, we pass the document string directly. For image hits, we load the file and pass it as a base64-encoded image to a vision-capable LLM. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 all accept image inputs in their chat APIs.

import base64
import os
from openai import OpenAI

llm = OpenAI()

def image_to_data_url(path: str) -> str:
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    ext = os.path.splitext(path)[1].lower().lstrip(".")
    mime = "jpeg" if ext in ("jpg", "jpeg") else ext
    return f"data:image/{mime};base64,{b64}"

def build_messages(query: str, hits: list[dict]) -> list[dict]:
    content = [{"type": "text", "text": f"User question: {query}\n\nUse the following retrieved context to answer. Cite whether each piece of evidence is text or an image."}]

    for i, hit in enumerate(hits):
        if hit["modality"] == "text":
            content.append({
                "type": "text",
                "text": f"[Text evidence {i+1}]: {hit['payload']}"
            })
        else:
            data_url = image_to_data_url(hit["image_path"])
            content.append({
                "type": "image_url",
                "image_url": {"url": data_url}
            })
            content.append({
                "type": "text",
                "text": f"[Image evidence {i+1}]"
            })

    content.append({"type": "text", "text": "Answer the user question now."})
    return [{"role": "user", "content": content}]

Step 5: Generating the Final Answer

With the messages assembled, call the vision LLM. The model receives the user's question alongside both text snippets and images, and it can reason across all of them to produce a grounded answer.

def answer(query: str, top_k: int = 5) -> str:
    hits = retrieve(query=query, top_k=top_k)
    messages = build_messages(query, hits)
    response = llm.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=512,
        temperature=0.2
    )
    return response.choices[0].message.content

print(answer("What does the interior of the Model X look like?"))

You can also support image-based queries by passing an image path instead of text:

def answer_with_image(image_path: str, question: str, top_k: int = 5) -> str:
    hits = retrieve(image_path=image_path, top_k=top_k)
    messages = build_messages(question, hits)
    response = llm.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_tokens=512
    )
    return response.choices[0].message.content

print(answer_with_image("./data/query_photo.jpg", "What product is this and what is its return policy?"))

Best Practices

Choose the Right CLIP Variant

ViT-B-32 is fine for prototyping, but retrieval quality scales with model size. For production, benchmark ViT-L-14 or ViT-H-14 on your own data. The larger models have 768- and 1024-dimensional embeddings respectively, which increases storage cost but meaningfully improves recall on fine-grained queries. If latency is critical, consider the smaller ViT-B-16, which is faster than B-32 on some hardware while being more accurate.

Normalize and Use Cosine Similarity

Always L2-normalize embeddings before storing them, and configure your vector database to use cosine similarity (or dot-product on normalized vectors, which is equivalent). Euclidean distance on un-normalized CLIP vectors produces noticeably worse rankings because CLIP was trained with cosine contrastive loss.

Handle Mixed-Modality Ranking Carefully

CLIP's text and image scores are not perfectly calibrated against each other. A text-to-text match and a text-to-image match with the same cosine distance do not necessarily represent the same confidence level. If you retrieve a mix of modalities and find one modality dominating, consider retrieving text and image candidates separately with their own top-k, then merging. You can also apply a small modality-specific bias term learned from a held-out evaluation set.

Store Originals, Not Just Embeddings

Embeddings are for retrieval; the generator needs the original content. For images, store the file path or a content-addressed reference and reload the full-resolution file at generation time. Do not try to reconstruct images from embeddings — that is not what CLIP is designed for. For text, store the full chunk plus metadata about its source document, page number, and surrounding context so the generator can cite provenance.

Chunk Text Thoughtfully

CLIP's text encoder has a token limit (77 tokens for the original OpenAI CLIP, though open_clip variants vary). Long paragraphs get truncated, which silently drops information. Keep text chunks under the token limit, and consider embedding both a short title-like summary and the full chunk, storing both vectors pointing to the same document. Retrieve on the summary, return the full chunk.

Evaluate with Multi-Modal Metrics

Standard RAG evaluation metrics like context precision and answer faithfulness still apply, but you should also measure cross-modal recall: given a text query, how often does the correct image appear in the top-k? Given an image query, how often does the correct text passage appear? Build a small labeled evaluation set with paired text-image queries to track this over time.

Consider a Reranker

CLIP embeddings are great for fast first-stage retrieval, but a cross-encoder reranker can improve precision significantly. For text-to-text reranking, models like bge-reranker work well. For cross-modal reranking, fine-tuned CLIP variants or specialized reranking models can reorder the top-20 candidates before sending them to the LLM. This two-stage pattern — cheap dense retrieval followed by expensive but accurate reranking — is the industry standard for high-quality RAG.

Watch Out for CLIP's Known Weaknesses

CLIP struggles with counting, spatial relationships ("left of," "above"), fine-grained attribute binding ("the red square, not the blue one"), and text rendered inside images. If your use case depends heavily on these, consider augmenting CLIP with OCR for in-image text, or use a more recent multi-modal embedding model like SigLIP or EVA-CLIP, which improve on several of these weaknesses.

Conclusion

Multi-Modal RAG with CLIP embeddings unlocks a class of applications that text-only RAG simply cannot serve — from visual product search to document QA over charts and diagrams to support bots that accept screenshots. The implementation is surprisingly approachable: one shared embedding model, one vector collection, and a vision-capable LLM at the end. The real work lies in the details — choosing the right CLIP variant, normalizing embeddings, handling modality calibration, chunking text within token limits, and evaluating cross-modal recall rigorously. Start with the pipeline shown here, benchmark on your own data, and iterate on the retrieval and reranking stages until the system reliably surfaces the right evidence regardless of whether it lives in words or pixels.