How to Implement Conversational Memory in RAG Applications

Retrieval-Augmented Generation (RAG) has become the go-to architecture for building LLM-powered applications that ground their answers in private or up-to-date data. However, a vanilla RAG pipeline is stateless: each user query is processed in isolation, with no awareness of what was said before. This is fine for one-shot lookups like "What is the return policy?" but breaks down the moment a user asks a follow-up such as "And how long does that take?" Conversational memory is the missing piece that turns a RAG system into a true chat experience.

What Is Conversational Memory in RAG?

Conversational memory is the mechanism by which an application stores and reuses prior turns of a conversation so the language model can interpret new queries in context. In a RAG setup, memory serves two distinct purposes: it helps the model understand the user's intent (resolving pronouns and references), and it can inform what gets retrieved from the vector store. Without it, a query like "Tell me more about the second one" is meaningless because the model has no idea what "the second one" refers to.

Concretely, conversational memory typically stores:

The user's previous questions
The assistant's previous answers
Optional metadata such as retrieved document IDs, timestamps, or tool calls
A condensed or summarized representation of older turns when the history grows long

Why Conversational Memory Matters

Stateless RAG pipelines suffer from several well-known failure modes when users interact conversationally:

Ambiguous follow-ups: Queries containing "it", "that", "the previous one", or "again" cannot be resolved.
Lost context across turns: A user refining a question ("Make it shorter", "Now in Spanish") loses the original subject.
Poor retrieval quality: The embedding of a short follow-up like "Why?" produces a weak vector that retrieves irrelevant documents.
Repetitive answers: The assistant re-explains concepts already covered, degrading the user experience.

By injecting conversation history into both the retrieval step and the generation step, you allow the system to resolve references, maintain continuity, and retrieve more relevant context. The result is an assistant that feels like it is actually listening.

Core Strategies for Implementing Memory

There are several patterns you can adopt, often combined. Each makes different trade-offs between latency, token cost, and accuracy.

1. Pass-Through History

The simplest approach is to append the full conversation history to the prompt sent to the LLM. This works for short sessions but quickly becomes expensive and can exceed the context window.

2. Query Rewriting / Standalone Question Generation

Before retrieving documents, you ask the LLM to rewrite the user's follow-up into a self-contained query using the conversation history. For example, "And how long does that take?" becomes "How long does the return process take?" given the previous turn discussed returns. This rewritten query is then embedded and used for retrieval.

3. Sliding Window Buffer

Keep only the last N turns in memory. Older turns are dropped. This keeps token usage bounded but loses long-range context.

4. Summarization Memory

Periodically summarize older turns into a compact paragraph and keep only the summary plus the most recent few turns. This preserves long-range context at a fraction of the token cost.

5. Entity Memory

Extract and track entities (people, products, dates) across the conversation in a structured store, then inject the relevant entity facts into the prompt. This is useful for domain-specific assistants.

Implementing Conversational Memory: A Practical Example

The example below uses Python with LangChain, a popular framework for building LLM applications. It demonstrates query rewriting for retrieval combined with a sliding window buffer for generation. The same patterns translate directly to LlamaIndex, Haystack, or a custom stack.

Prerequisites

Install the required packages and set your OpenAI API key:

pip install langchain langchain-openai langchain-community chromadb
export OPENAI_API_KEY="sk-your-key-here"

Building the Memory-Aware RAG Chain

First, set up the LLM, embeddings, and a small in-memory vector store for demonstration purposes:

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Seed a tiny knowledge base
docs = [
    Document(page_content="Our standard return window is 30 days from the delivery date.",
             metadata={"source": "policy.md"}),
    Document(page_content="Refunds are processed within 5 to 7 business days to the original payment method.",
             metadata={"source": "policy.md"}),
    Document(page_content="We ship to over 40 countries with delivery times ranging from 2 to 10 business days.",
             metadata={"source": "shipping.md"}),
]
vectorstore = Chroma.from_documents(docs, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

Next, create a query-rewriting step. This is the heart of conversational retrieval: it converts a context-dependent follow-up into a standalone query that retrieves well:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

rewrite_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "Given the chat history and the latest user question, "
     "rewrite it as a standalone question that can be understood "
     "without the chat history. Do NOT answer the question, just "
     "rewrite it. If it is already standalone, return it unchanged."),
    ("placeholder", "{history}"),
    ("human", "{question}"),
])

rewrite_chain = rewrite_prompt | llm | StrOutputParser()

Now build the final answer-generation prompt, which includes both the retrieved documents and the conversation history:

answer_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "You are a helpful assistant. Use the following retrieved "
     "context to answer the user's question. If the context does "
     "not contain the answer, say you don't know.\n\nContext:\n{context}"),
    ("placeholder", "{history}"),
    ("human", "{question}"),
])

Define a simple in-memory store for the conversation history. In production you would back this with Redis, Postgres, or a dedicated memory service keyed by session ID:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

# session_id -> history
histories = {}

def get_history(session_id: str) -> InMemoryChatMessageHistory:
    if session_id not in histories:
        histories[session_id] = InMemoryChatMessageHistory()
    return histories[session_id]

Now assemble the full chain. The chain first rewrites the question, retrieves documents using the rewritten query, formats the context, and finally generates the answer:

from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from operator import itemgetter

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

# Step 1: rewrite the question using history
# Step 2: retrieve with the rewritten question
# Step 3: answer using retrieved docs + history
rag_chain = (
    {
        "standalone_question": rewrite_chain,
        "history": itemgetter("history"),
        "original_question": itemgetter("question"),
    }
    | {
        "context": itemgetter("standalone_question") | retriever | format_docs,
        "history": itemgetter("history"),
        "question": itemgetter("original_question"),
    }
    | answer_prompt
    | llm
)

Wrap the chain with message-history management so prior turns are automatically loaded and new turns are automatically saved:

conversational_rag = RunnableWithMessageHistory(
    rag_chain,
    get_history,
    input_messages_key="question",
    history_messages_key="history",
)

Running a Multi-Turn Conversation

Now you can simulate a user asking a question and then a follow-up that depends on the first turn:

config = {"configurable": {"session_id": "user-123"}}

# Turn 1
answer1 = conversational_rag.invoke(
    {"question": "What is your return policy?"},
    config=config,
)
print("Assistant:", answer1.content)

# Turn 2: a follow-up that only makes sense with memory
answer2 = conversational_rag.invoke(
    {"question": "And how long does that take?"},
    config=config,
)
print("Assistant:", answer2.content)

# Turn 3: another reference-dependent question
answer3 = conversational_rag.invoke(
    {"question": "Does that apply to international orders too?"},
    config=config,
)
print("Assistant:", answer3.content)

Without memory, turn 2 ("And how long does that take?") would retrieve nothing useful because the embedding of that sentence is generic. With query rewriting, the chain converts it into something like "How long does the return refund process take?" which retrieves the refund-timing document. Turn 3 similarly gets rewritten to reference international shipping and refunds.

Adding Summarization for Long Conversations

Sliding windows drop older context entirely. For long-running sessions such as support agents or tutoring bots, you may want a summarization layer. The idea is to keep the last K turns verbatim and replace everything older with a running summary that is updated every few turns.

from langchain_core.messages import SystemMessage, HumanMessage, AIMessage

summarize_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "Progressively summarize the conversation. Provide a concise "
     "running summary of the most important facts, entities, and "
     "user goals so far. Incorporate the new lines below into the "
     "existing summary.\n\nCurrent summary:\n{summary}\n\nNew lines:\n{new_lines}"),
])

summarize_chain = summarize_prompt | llm | StrOutputParser()

class SummaryMemory:
    def __init__(self, keep_last: int = 4):
        self.summary = ""
        self.buffer = []  # list of (role, content)
        self.keep_last = keep_last

    def add_turn(self, role: str, content: str):
        self.buffer.append((role, content))
        if len(self.buffer) > self.keep_last:
            # Move older turns into the summary
            to_summarize = self.buffer[:-self.keep_last]
            self.buffer = self.buffer[-self.keep_last:]
            new_lines = "\n".join(f"{r}: {c}" for r, c in to_summarize)
            self.summary = summarize_chain.invoke(
                {"summary": self.summary, "new_lines": new_lines}
            )

    def as_messages(self):
        msgs = []
        if self.summary:
            msgs.append(SystemMessage(content=f"Summary so far: {self.summary}"))
        for role, content in self.buffer:
            if role == "user":
                msgs.append(HumanMessage(content=content))
            else:
                msgs.append(AIMessage(content=content))
        return msgs

You would then feed as_messages() into the history slot of your chain instead of the raw buffer. This keeps token usage roughly constant no matter how long the conversation runs.

Best Practices

Always rewrite before retrieving. Embedding a raw follow-up produces poor neighbors. A cheap rewrite step dramatically improves retrieval relevance.
Scope memory by session ID. Never share history across users. Use a stable identifier per browser session or authenticated user.
Persist memory outside the process. In-memory dicts vanish on restart. Use Redis, Postgres, or a managed store for durability.
Cap token usage. Combine a sliding window with summarization so a 200-turn chat does not blow your context window or budget.
Store retrieved document IDs, not just text. This lets you deduplicate context across turns and avoid re-retrieving the same chunks.
Log the rewritten query. It is invaluable for debugging retrieval failures and for evaluating quality.
Let users reset. Provide a "clear conversation" affordance that wipes the session history, both for privacy and for usability.
Test with multi-turn eval sets. Single-turn QA benchmarks hide memory bugs. Build eval scenarios with explicit follow-ups and reference-dependent questions.
Consider privacy and retention. Decide how long histories live, who can read them, and how they are encrypted at rest.

Conclusion

Conversational memory is what separates a search box with an LLM attached from a genuine assistant. By layering query rewriting for retrieval, a sliding window for short-term context, and summarization for long sessions, you can build RAG applications that handle follow-ups, references, and refinements gracefully. The implementation is not conceptually complex, but the details matter: scope memory per session, persist it durably, cap token growth, and always evaluate with multi-turn scenarios. Get these foundations right, and your users will feel like they are talking to something that actually remembers what they just said.