Introduction to Handling PDF Tables and Images in RAG Pipelines
Retrieval-Augmented Generation (RAG) pipelines have become the standard for building question-answering systems over private data. However, a significant portion of enterprise data exists in PDF format, which often contains complex layouts, tables, and images. Standard text extraction tools treat PDFs as flat streams of text, completely ignoring the structural and visual context. Handling PDF tables and images in RAG pipelines refers to the process of accurately identifying, extracting, and converting these non-text elements into searchable, retrievable text or vector representations.
This matters because tables often contain the most critical quantitative data—financial figures, technical specifications, and schedules—while images and charts provide visual context that text alone cannot convey. If your RAG pipeline ignores these elements, the Large Language Model (LLM) will suffer from blind spots, leading to incomplete or hallucinated answers. By properly parsing tables and images, you ensure your RAG system has a holistic understanding of the document.
Understanding the Challenges of PDF Parsing
PDFs are designed for presentation, not data extraction. The text is positioned using absolute coordinates, meaning the reading order can be scrambled. Tables are essentially just text boxes drawn close together with border lines, making it difficult for basic extractors to distinguish between a table and a multi-column paragraph. Images are embedded as binary streams, completely invisible to text-based RAG retrievers. To solve this, developers must use specialized libraries and multimodal approaches.
How to Extract and Process PDF Tables
To extract tables, you need a library that understands spatial relationships. pdfplumber is an excellent Python library for this task. It can detect table boundaries and extract the data into structured formats like lists of lists, which can then be converted into Markdown or CSV for better LLM comprehension.
Extracting Tables with pdfplumber
Below is a practical example of how to extract tables from a PDF and format them as Markdown strings, which LLMs parse exceptionally well.
import pdfplumber
def extract_tables_to_markdown(pdf_path):
table_markdowns = []
with pdfplumber.open(pdf_path) as pdf:
for page_num, page in enumerate(pdf.pages):
# Extract tables from the current page
tables = page.extract_tables()
for table_idx, table in enumerate(tables):
if not table:
continue
# Convert table to Markdown format
markdown_table = "| " + " | ".join(table[0]) + " |\n"
markdown_table += "| " + " | ".join(["---"] * len(table[0])) + " |\n"
for row in table[1:]:
# Handle potential None values in rows
clean_row = [str(cell) if cell is not None else "" for cell in row]
markdown_table += "| " + " | ".join(clean_row) + " |\n"
table_markdowns.append({
"page": page_num + 1,
"table_index": table_idx + 1,
"content": markdown_table
})
return table_markdowns
# Usage
extracted_tables = extract_tables_to_markdown("financial_report.pdf")
for t in extracted_tables:
print(f"Page {t['page']}, Table {t['table_index']}:\n{t['content']}\n")
How to Extract and Process PDF Images
Images and charts require a multimodal approach. First, you extract the image binaries from the PDF using a library like PyMuPDF (imported as fitz). Then, you pass these images to a Vision-Language Model (VLM) like OpenAI's GPT-4o or Anthropic's Claude 3 to generate a detailed text description. This description is then injected into your RAG pipeline as standard text.
Extracting Images and Generating Descriptions
The following code demonstrates how to extract images from a PDF and use the OpenAI API to generate descriptive summaries.
import fitz # PyMuPDF
import base64
from openai import OpenAI
client = OpenAI()
def get_image_description(image_bytes):
base64_image = base64.b64encode(image_bytes).decode('utf-8')
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image or chart in detail, focusing on any data, trends, or key visual elements."},
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_image}"
}
}
]
}
],
max_tokens=300
)
return response.choices[0].message.content
def extract_and_describe_images(pdf_path):
image_descriptions = []
doc = fitz.open(pdf_path)
for page_num in range(len(doc)):
page = doc[page_num]
image_list = page.get_images(full=True)
for img_idx, img in enumerate(image_list):
xref = img[0]
base_image = doc.extract_image(xref)
image_bytes = base_image["image"]
# Get description from VLM
description = get_image_description(image_bytes)
image_descriptions.append({
"page": page_num + 1,
"image_index": img_idx + 1,
"description": description
})
return image_descriptions
# Usage
image_data = extract_and_describe_images("technical_spec.pdf")
for img in image_data:
print(f"Page {img['page']}, Image {img['image_index']} Description: {img['description']}\n")
Integrating Extracted Data into a RAG Pipeline
Once you have extracted the standard text, the Markdown tables, and the image descriptions, you must unify them into a single document structure. A common strategy is to insert the table Markdown and image descriptions directly into the text stream at the approximate location where they appeared on the page. This preserves the local context.
Chunking and Embedding Strategies
When chunking the document for the vector database, ensure that tables and image descriptions do not get split awkwardly. You can use LangChain's document loaders and splitters to manage this.
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
def build_rag_documents(raw_text, tables, images):
# In a real scenario, you would interleave these based on page coordinates.
# For simplicity, we append them to the end of the page text.
full_content = raw_text + "\n\n"
for t in tables:
full_content += f"**Extracted Table (Page {t['page']}):**\n{t['content']}\n\n"
for img in images:
full_content += f"**Image Description (Page {img['page']}):**\n{img['description']}\n\n"
# Create a LangChain Document
doc = Document(page_content=full_content, metadata={"source": "complex_pdf.pdf"})
# Split text while trying to keep tables and descriptions intact
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents([doc])
return chunks
# Usage
# chunks = build_rag_documents(extracted_text, extracted_tables, image_data)
# print(f"Created {len(chunks)} chunks for the vector database.")
Best Practices for PDF RAG Pipelines
- Use Markdown for Tables: LLMs are heavily trained on Markdown. Converting extracted tables to Markdown format significantly improves the model's ability to understand and reason over the data.
- Preserve Metadata: Always attach metadata to your chunks, such as page number, source file name, and content type (e.g., "text", "table", "image"). This allows for metadata filtering during retrieval.
- Handle Scanned PDFs: If your PDFs are scanned images rather than digitally generated, text extraction will fail. Implement an OCR step (using Tesseract or AWS Textract) before attempting table or text extraction.
- Use Multimodal Embeddings: Instead of converting images to text, consider using multimodal embedding models (like CLIP) to embed the image directly alongside text embeddings. This preserves visual nuances that a VLM text summary might miss.
- Implement Fallback Logic: Table extraction is rarely 100% accurate. If a table extraction library returns empty or malformed data, fall back to passing the entire page as an image to a VLM to extract the table context.
Conclusion
Handling tables and images in PDF-based RAG pipelines is essential for building robust, enterprise-grade AI applications. By moving beyond basic text extraction and utilizing spatial parsing libraries like pdfplumber alongside multimodal vision models, developers can capture the full context of complex documents. While the process requires more computational overhead and careful integration, the resulting accuracy and completeness of the LLM's answers make it a worthwhile investment. By following the techniques and best practices outlined above, you can ensure your RAG system sees the whole picture, not just the plain text.