← Back to DevBytes
Developer Tutorial July 20, 2026 9 min read

Building RAG Over PDFs: Parsing, Tables, and Images

What is RAG Over PDFs and Why It's Challenging

Retrieval Augmented Generation (RAG) allows large language models to ground their answers in your own documents. When those documents are PDFs, a whole new layer of complexity emerges. PDFs are not simply text files; they can contain mixed content — paragraphs, tables, images, headers, footers, and multi-column layouts — often with no logical reading order exposed to the parser.

Building a RAG pipeline that simply extracts plain text from PDFs ignores tables and images, leading to incomplete or misleading answers. This tutorial walks you through building a robust PDF RAG system that parses text, tables, and images, processes each type into LLM-friendly formats, and integrates them into a unified retrieval pipeline.

The Core Challenge: PDFs Are Not Plain Text

A PDF stores drawing instructions (text positions, vector graphics, embedded images) rather than a clean sequence of words. Extracting content requires reconstructing the logical flow. Libraries like PyPDF2 or pdfplumber attempt to group characters into lines and paragraphs, but they often miss tables (which appear as scattered text snippets) and ignore images entirely. For a RAG system to answer questions like “What was the revenue in Q3?” or “Explain the diagram on page 5,” we must capture and index tables and images alongside text.

Why Parsing Tables and Images Matters for RAG

When a user asks a question, the retrieval step searches for the most relevant chunks. If a financial table is stored only as disjointed text lines, a semantic embedding of “revenue 2023” might fail to match “Q3 revenue $1.2M”. Similarly, an image showing a critical architecture diagram will be invisible to a text-only RAG. By converting tables into structured text (CSV, Markdown) and images into descriptive captions or embedded summaries, we enable the system to find and present exactly the right evidence.

Step-by-Step: Building a RAG Pipeline for PDFs

Below we build a complete pipeline using Python. We'll cover extraction, preprocessing, chunking, embedding, indexing, and the final retrieval + generation loop. All code examples are self-contained and ready to adapt.

1. Extracting Text, Tables, and Images

We use pdfplumber for text and table extraction because it returns tables as lists of rows while also giving page-level text. For images, we'll use PyMuPDF (fitz) which reliably extracts images and their bounding boxes. 


import pdfplumber
import fitz  # PyMuPDF
import os
from PIL import Image
import io

def extract_page_content(pdf_path, page_number, output_image_dir="images"):
    # --- pdfplumber for text and tables ---
    with pdfplumber.open(pdf_path) as pdf:
        page = pdf.pages[page_number]
        text = page.extract_text()
        tables = page.extract_tables()

    # --- PyMuPDF for images ---
    doc = fitz.open(pdf_path)
    page = doc[page_number]
    images = []
    for img_index, img in enumerate(page.get_images(full=True)):
        xref = img[0]
        base_image = doc.extract_image(xref)
        image_bytes = base_image["image"]
        ext = base_image["ext"]
        pil_image = Image.open(io.BytesIO(image_bytes))
        # save for later processing
        img_name = f"page{page_number+1}_img{img_index+1}.{ext}"
        pil_image.save(os.path.join(output_image_dir, img_name))
        images.append(img_name)
    doc.close()

    return {
        "text": text,
        "tables": tables,
        "images": images
    }

# Example usage
result = extract_page_content("report.pdf", 0)
print("Text snippet:", result["text"][:200])
print("Tables found:", len(result["tables"]))
print("Images extracted:", result["images"])

pdfplumber returns tables as nested lists (e.g., [['Name','Value'], ['Alice','42']]). This structured form is key for the next step.

2. Preprocessing Tables for LLM Consumption

Raw nested lists are not ideal for embedding or prompting. We convert tables into a lightweight markup like Markdown, or a CSV string, that preserves row/column relationships. Markdown tables are well understood by modern LLMs. 


def table_to_markdown(table):
    """Convert pdfplumber table (list of lists) into Markdown table string."""
    if not table:
        return ""
    # clean: remove None values, replace empty with ""
    cleaned = []
    for row in table:
        cleaned_row = [cell if cell is not None else "" for cell in row]
        cleaned.append(cleaned_row)
    if not cleaned:
        return ""
    
    header = cleaned[0]
    # create separator line
    sep = ["---"] * len(header)
    body = cleaned[1:]
    
    lines = []
    lines.append("| " + " | ".join(header) + " |")
    lines.append("| " + " | ".join(sep) + " |")
    for row in body:
        lines.append("| " + " | ".join(row) + " |")
    return "\n".join(lines)

# Usage
tables = result["tables"]
markdown_tables = [table_to_markdown(t) for t in tables]
print(markdown_tables[0])

Alternatively, you can convert to CSV if your embedding model works better with comma-separated formats. The goal is a string representation that preserves table semantics.

3. Handling Images: OCR and Summarization

Extracted images may be charts, diagrams, or scanned text. We apply OCR (if needed) and then generate a textual description using a multimodal model. Here we use a lightweight pipeline: first try OCR with pytesseract for scanned images, then use a captioning model like Salesforce/blip-image-captioning-base for visual understanding. 


from PIL import Image
import pytesseract
from transformers import pipeline

def describe_image(image_path, use_ocr=True):
    """Generate a text description of an image for RAG indexing."""
    img = Image.open(image_path)
    descriptions = []

    # OCR for any embedded text
    if use_ocr:
        try:
            ocr_text = pytesseract.image_to_string(img)
            if ocr_text.strip():
                descriptions.append(f"OCR text: {ocr_text.strip()}")
        except:
            pass

    # Visual captioning
    captioner = pipeline("image-to-text", model="Salesforce/blip-image-captioning-base")
    caption = captioner(img)[0]["generated_text"]
    descriptions.append(f"Caption: {caption}")
    
    return " | ".join(descriptions)

# Process extracted images
image_descriptions = []
for img_name in result["images"]:
    img_path = os.path.join("images", img_name)
    desc = describe_image(img_path)
    image_descriptions.append({"image": img_name, "description": desc})
    print(f"{img_name} -> {desc}")

For a production system, you might use a vision-enabled LLM (GPT-4 Vision, Gemini Pro Vision) via API for higher quality captions. The above gives a local, free alternative.

4. Chunking with Context Preservation

We must chunk text and integrate table/image descriptions into the same semantic blocks. A naive split of text alone will break the association between a table and the paragraph that explains it. Instead, we construct chunks that merge nearby elements from the same page. 


from langchain.text_splitter import RecursiveCharacterTextSplitter

def build_document_chunks(page_data, page_number, chunk_size=500, overlap=50):
    """
    Combine text, table markdowns, and image descriptions into a single
    page-level string, then split into overlapping chunks with metadata.
    """
    # Assemble page content with delimiters
    parts = []
    if page_data["text"]:
        parts.append(page_data["text"])
    for md_table in page_data.get("markdown_tables", []):
        parts.append("[TABLE]\n" + md_table)
    for img_desc in page_data.get("image_descriptions", []):
        parts.append("[IMAGE]\n" + img_desc["description"])
    
    full_page = "\n\n".join(parts)
    
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", " ", ""]
    )
    chunks = splitter.split_text(full_page)
    
    # Attach metadata
    return [
        {"content": chunk, "metadata": {"page": page_number + 1, "source": "report.pdf"}}
        for chunk in chunks
    ]

# Example
page_data = {
    "text": result["text"],
    "markdown_tables": markdown_tables,
    "image_descriptions": image_descriptions
}
chunks = build_document_chunks(page_data, page_number=0)
print(f"Produced {len(chunks)} chunks from page 1")
print("First chunk:", chunks[0]["content"][:150])

The [TABLE] and [IMAGE] markers help the LLM later understand the provenance of each piece. Overlapping chunks ensure a table that falls near a chunk boundary isn't orphaned.

5. Embedding and Indexing

We embed each chunk using a sentence-transformer model and store them in a vector database. Here we use ChromaDB for simplicity, but FAISS or Pinecone work similarly. 


from chromadb import Client
from chromadb.utils import embedding_functions

# Initialize ChromaDB with an embedding function
client = Client()
collection_name = "pdf_rag_collection"
embedding_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-MiniLM-L6-v2"
)

# Create or get collection
try:
    collection = client.get_collection(collection_name, embedding_function=embedding_fn)
except:
    collection = client.create_collection(collection_name, embedding_function=embedding_fn)

# Add chunks (this example does all pages; you'd loop over pages)
ids = [f"page1_chunk{i}" for i in range(len(chunks))]
contents = [chunk["content"] for chunk in chunks]
metadatas = [chunk["metadata"] for chunk in chunks]

collection.add(
    ids=ids,
    documents=contents,
    metadatas=metadatas
)
print(f"Indexed {len(contents)} chunks.")

If you have many PDFs, loop over each file and page, extract content, build chunks, and add to the same collection. The embedding model must be consistent across all indexing and querying.

6. Retrieval and Augmented Generation

Finally, we query the index, retrieve the most relevant chunks, and feed them into an LLM with a prompt that instructs the model to interpret tables and images properly. 


def rag_query(question, collection, llm_model="gpt-3.5-turbo", top_k=3):
    # Retrieve top-k chunks
    results = collection.query(
        query_texts=[question],
        n_results=top_k
    )
    documents = results["documents"][0]  # list of strings
    metadatas = results["metadatas"][0]

    # Build context string with source markers
    context_parts = []
    for i, doc in enumerate(documents):
        source_page = metadatas[i].get("page", "unknown")
        context_parts.append(f"[Source Page {source_page}]\n{doc}")
    context = "\n\n".join(context_parts)

    # Construct prompt
    prompt = f"""You are an assistant that answers questions using the provided document excerpts. 
The excerpts may contain [TABLE] and [IMAGE] markers followed by structured data. 
When using table data, explain it clearly. 
If an image caption is relevant, mention what the image depicts.

Context:
{context}

Question: {question}

Answer (be concise and base strictly on the context):"""

    # Call LLM (replace with your actual API call)
    # Here we simulate with a print statement
    print("=== PROMPT ===")
    print(prompt)
    # response = openai.ChatCompletion.create(
    #     model=llm_model,
    #     messages=[{"role": "user", "content": prompt}]
    # )
    # return response.choices[0].message.content
    return "This is a placeholder response."

# Example query
rag_query("What was the total revenue in 2023 according to the table?", collection)

The prompt explicitly mentions [TABLE] and [IMAGE] markers so the LLM knows how to interpret those sections. In practice, you'd replace the placeholder with your preferred LLM API (OpenAI, Anthropic, or a local model).

Best Practices for Production-Ready PDF RAG

Conclusion

Building a RAG system over PDFs is not simply dumping text into a vector store. Tables and images contain critical information that plain text extraction discards. By extracting tables as structured Markdown, generating textual descriptions for images, and merging everything into coherent chunks, you preserve the document's full semantic richness. The pipeline we built — using pdfplumber, PyMuPDF, a captioning model, and a vector database — provides a solid foundation. Extend it with production-grade parsers, hybrid search, and careful metadata tracking, and you'll have a RAG system that truly understands your PDFs, tables, diagrams and all.

— Ad —

Google AdSense will appear here after approval

← Back to all articles