Introduction to Agent Memory Architectures
Memory is one of the most critical components of any autonomous agent. Without it, an LLM-based agent is stateless — every interaction begins from scratch, and the agent cannot learn, recall, or build on prior context. LlamaIndex provides a flexible, modular memory system that lets developers build agents capable of remembering conversations, facts, and long-term knowledge across sessions.
In this tutorial, you'll learn what agent memory architectures are, why they matter, and how to implement them using LlamaIndex's memory primitives. We'll cover short-term conversation buffers, long-term vector-backed memory, memory blocks, and custom architectures that combine multiple memory strategies.
What Is Agent Memory?
Agent memory refers to the mechanisms an agent uses to store, retrieve, and use information across interactions. Unlike a simple chatbot that only sees the current message, a memory-enabled agent can:
- Recall facts discussed earlier in a conversation
- Remember user preferences and personalization details
- Retrieve relevant historical context when answering new questions
- Learn from past interactions to improve future responses
- Maintain continuity across multiple sessions and devices
Memory in LlamaIndex is built around the idea of BaseMemory — an abstract interface that any memory implementation must satisfy. This abstraction lets you swap memory strategies without rewriting your agent logic.
Why Memory Matters for Agents
Without memory, agents face several fundamental limitations. First, the context window of any LLM is finite. Even with models supporting 128K or 1M tokens, you cannot fit an entire user history into a single prompt indefinitely. Memory systems let you selectively retrieve only what's relevant.
Second, agents that operate over long horizons — research assistants, coding agents, customer support bots — need to maintain state. A coding agent that forgets the project structure after every turn is useless. Memory provides the persistence layer that makes agents genuinely useful.
Third, memory enables personalization. By remembering user preferences, past corrections, and interaction patterns, agents can tailor responses in ways that stateless systems cannot.
Types of Memory Architectures
Memory architectures generally fall into several categories, each with different trade-offs:
Short-Term Memory
Short-term memory holds the most recent conversation turns. It's typically implemented as a sliding window buffer that keeps the last N messages. This is fast, simple, and works well for focused conversations, but it forgets anything outside the window.
Long-Term Memory
Long-term memory persists information across sessions. The most common implementation uses vector embeddings stored in a vector database. When the agent needs context, it retrieves the most semantically relevant memories. This scales to large histories but requires embedding computation and retrieval latency.
Summary Memory
Summary memory compresses older conversation history into a running summary. Instead of keeping every message, the agent periodically summarizes what happened and stores the summary. This preserves key information while keeping token usage bounded.
Hybrid Memory
Hybrid architectures combine multiple strategies — for example, keeping recent messages in a buffer while summarizing older ones and retrieving relevant facts from a vector store. LlamaIndex's memory blocks make these combinations straightforward.
LlamaIndex Memory Modules Overview
LlamaIndex provides several memory primitives in the llama_index.core.memory module:
ChatMemoryBuffer— a token-bounded buffer for conversation historyVectorMemory— vector-backed memory for semantic retrievalMemory— the newer high-level memory interface with blocksStaticMemoryBlock,DynamicMemoryBlock,VectorMemoryBlock— composable memory blocks
Let's start by installing LlamaIndex and setting up a basic environment.
Setting Up Your Environment
Install LlamaIndex with the core dependencies. We'll also install a vector store and embedding model for long-term memory examples:
pip install llama-index llama-index-vector-stores-chroma llama-index-embeddings-openai
Set your OpenAI API key as an environment variable:
import os
os.environ["OPENAI_API_KEY"] = "sk-your-key-here"
Now let's build our first memory-enabled agent.
Short-Term Memory with ChatMemoryBuffer
The simplest memory architecture is a conversation buffer. ChatMemoryBuffer stores messages and automatically trims when the token count exceeds a configured limit. Here's a complete example:
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.llms.openai import OpenAI
# Create a memory buffer with a token limit
memory = ChatMemoryBuffer.from_defaults(
token_limit=3000,
chat_store=None, # in-memory by default
)
# Initialize the LLM
llm = OpenAI(model="gpt-4o-mini", temperature=0.7)
# Simulate a conversation
messages = [
"Hi, I'm Sarah and I work as a data scientist at a healthcare startup.",
"What tools should I learn for building RAG applications?",
"Can you remind me what my role is?",
]
for user_msg in messages:
memory.put(memory.chat_store_key, {"role": "user", "content": user_msg})
print(f"User: {user_msg}")
# Get current memory state
chat_history = memory.get(memory.chat_store_key)
response = llm.chat(chat_history)
memory.put(
memory.chat_store_key,
{"role": "assistant", "content": response.message.content},
)
print(f"Agent: {response.message.content}\n")
The agent remembers Sarah's role because it's stored in the buffer. However, once the conversation exceeds 3000 tokens, older messages get trimmed. For longer conversations, you need a more sophisticated approach.
Using ChatMemoryBuffer with Agents
In practice, you'll usually pass memory to a LlamaIndex agent rather than managing messages manually:
from llama_index.core.agent import FunctionAgent
from llama_index.core.tools import FunctionTool
from llama_index.core.memory import ChatMemoryBuffer
def get_weather(location: str) -> str:
"""Get the weather for a location."""
return f"The weather in {location} is sunny and 72°F."
weather_tool = FunctionTool.from_defaults(fn=get_weather)
memory = ChatMemoryBuffer.from_defaults(token_limit=4000)
agent = FunctionAgent(
tools=[weather_tool],
llm=OpenAI(model="gpt-4o-mini"),
memory=memory,
system_prompt="You are a helpful assistant. Remember user details.",
)
# Run a multi-turn conversation
response = await agent.run(user_msg="I'm planning a trip to Paris.")
print(response)
response = await agent.run(user_msg="What's the weather like there?")
print(response)
response = await agent.run(user_msg="Where did I say I was going?")
print(response)
The agent remembers the Paris context across turns because ChatMemoryBuffer maintains the conversation history between calls.
Long-Term Memory with Vector Stores
For memory that persists across sessions and scales to large histories, use vector-backed memory. Each message or fact gets embedded and stored in a vector database. At query time, the agent retrieves the most relevant memories.
Here's a complete implementation using Chroma as the vector store:
import chromadb
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core.embeddings import resolve_embed_model
from llama_index.core.schema import Document
# Set up Chroma
db = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = db.get_or_create_collection("agent_memory")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# Use a local embedding model for simplicity
embed_model = resolve_embed_model("local:BAAI/bge-small-en-v1.5")
# Create an index for long-term memory
memory_index = VectorStoreIndex.from_vector_store(
vector_store=vector_store,
embed_model=embed_model,
)
# Store a fact in long-term memory
def remember(fact: str, metadata: dict = None):
"""Store a fact in long-term memory."""
doc = Document(text=fact, metadata=metadata or {})
memory_index.insert(doc)
print(f"Remembered: {fact}")
# Retrieve relevant memories
def recall(query: str, top_k: int = 3):
"""Retrieve relevant memories for a query."""
retriever = memory_index.as_retriever(similarity_top_k=top_k)
results = retriever.retrieve(query)
return [r.get_content() for r in results]
# Example usage
remember("User's name is Sarah Chen", {"type": "identity"})
remember("User works as a data scientist at HealthTech Inc.", {"type": "profile"})
remember("User prefers Python and uses pandas daily", {"type": "preference"})
remember("User is building a RAG system for medical literature", {"type": "project"})
# Recall relevant memories
memories = recall("What programming language does the user like?")
for m in memories:
print(f"- {m}")
This gives your agent a persistent knowledge base that survives restarts. The chroma_db directory stores everything on disk.
Combining Short-Term and Long-Term Memory
The most effective agents combine both memory types. Short-term memory handles the current conversation flow, while long-term memory provides background context. Here's how to build a hybrid memory agent:
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.agent import FunctionAgent
from llama_index.llms.openai import OpenAI
class HybridMemoryAgent:
def __init__(self, memory_index, llm=None):
self.short_term = ChatMemoryBuffer.from_defaults(token_limit=3000)
self.long_term_index = memory_index
self.llm = llm or OpenAI(model="gpt-4o-mini", temperature=0.3)
def _build_context(self, user_msg: str) -> str:
"""Retrieve relevant long-term memories for the current query."""
retriever = self.long_term_index.as_retriever(similarity_top_k=3)
results = retriever.retrieve(user_msg)
if not results:
return ""
memory_text = "\n".join([f"- {r.get_content()}" for r in results])
return f"Relevant memories from past interactions:\n{memory_text}"
async def chat(self, user_msg: str) -> str:
# Retrieve long-term context
long_term_context = self._build_context(user_msg)
# Build the full message list
from llama_index.core.llms import ChatMessage, MessageRole
messages = []
# Add system prompt with long-term context
system_content = (
"You are a helpful assistant with memory capabilities. "
"Use the provided memories when relevant.\n\n"
)
if long_term_context:
system_content += long_term_context
messages.append(ChatMessage(role=MessageRole.SYSTEM, content=system_content))
# Add short-term conversation history
history = self.short_term.get(self.short_term.chat_store_key)
messages.extend(history)
# Add current user message
messages.append(ChatMessage(role=MessageRole.USER, content=user_msg))
# Generate response
response = self.llm.chat(messages)
# Store in short-term memory
self.short_term.put(
self.short_term.chat_store_key,
{"role": "user", "content": user_msg},
)
self.short_term.put(
self.short_term.chat_store_key,
{"role": "assistant", "content": response.message.content},
)
return response.message.content
def remember_long_term(self, fact: str):
"""Persist a fact to long-term memory."""
from llama_index.core.schema import Document
self.long_term_index.insert(Document(text=fact))
# Usage
agent = HybridMemoryAgent(memory_index)
# Store some long-term facts
agent.remember_long_term("User's name is Sarah Chen")
agent.remember_long_term("User is allergic to shellfish")
agent.remember_long_term("User's birthday is March 15")
# Chat with the agent
import asyncio
async def main():
response = await agent.chat("Hi! Do you know anything about me?")
print(f"Agent: {response}\n")
response = await agent.chat("What should I avoid eating?")
print(f"Agent: {response}\n")
response = await agent.chat("When is my birthday?")
print(f"Agent: {response}\n")
asyncio.run(main())
This hybrid approach gives you the best of both worlds: immediate conversational context from the buffer and deep background knowledge from the vector store.
Memory Blocks in LlamaIndex
LlamaIndex's newer Memory class introduces the concept of memory blocks — composable units that each handle a specific type of memory. This is a more structured approach than manually combining buffers and vector stores.
from llama_index.core.memory import Memory
from llama_index.core.memory.blocks import (
StaticMemoryBlock,
DynamicMemoryBlock,
VectorMemoryBlock,
)
from llama_index.core.agent import FunctionAgent
from llama_index.llms.openai import OpenAI
# A static block holds fixed context that never changes
static_block = StaticMemoryBlock(
name="user_profile",
static_content=(
"User profile:\n"
"- Name: Sarah Chen\n"
"- Role: Data Scientist at HealthTech Inc.\n"
"- Preferred language: Python\n"
"- Timezone: PST"
),
)
# A dynamic block extracts relevant info from conversation
dynamic_block = DynamicMemoryBlock(
name="extracted_facts",
llm=OpenAI(model="gpt-4o-mini"),
extraction_prompt=(
"Extract key facts, preferences, and decisions from the "
"conversation. Return them as a concise list. If nothing "
"notable, return an empty string."
),
max_tokens=500,
)
# A vector block retrieves semantically relevant past context
vector_block = VectorMemoryBlock(
name="relevant_history",
vector_store=vector_store,
embed_model=embed_model,
retrieval_prompt="Retrieve memories relevant to the current query.",
top_k=3,
)
# Combine blocks into a Memory instance
memory = Memory.from_blocks(
session_default_blocks=["user_profile", "extracted_facts", "relevant_history"],
token_limit=6000,
)
# Create an agent with the memory
agent = FunctionAgent(
tools=[],
llm=OpenAI(model="gpt-4o-mini"),
memory=memory,
system_prompt=(
"You are a helpful assistant with persistent memory. "
"Use the provided memory blocks to personalize responses."
),
)
# Run conversations
import asyncio
async def run():
response = await agent.run(user_msg="Hi, I need help with my RAG project.")
print(response)
response = await agent.run(user_msg="Can you suggest some evaluation metrics?")
print(response)
# The dynamic block will have extracted facts about the RAG project
# The vector block will retrieve any stored memories about RAG
asyncio.run(run())
Memory blocks are powerful because each block independently manages its own content and token budget. The Memory class orchestrates them, inserting the right content into the prompt at the right time.
Building a Custom Memory Architecture
Sometimes the built-in options don't fit your use case. You can implement a custom memory class by subclassing BaseMemory. Here's an example of a memory system that maintains a running summary:
from llama_index.core.memory import BaseMemory
from llama_index.core.llms import ChatMessage, MessageRole
from llama_index.llms.openai import OpenAI
from typing import List, Dict, Any, Optional
class SummaryMemory(BaseMemory):
"""Memory that summarizes old messages while keeping recent ones."""
def __init__(
self,
llm: Optional[OpenAI] = None,
recent_messages_limit: int = 6,
max_summary_tokens: int = 500,
):
self.llm = llm or OpenAI(model="gpt-4o-mini", temperature=0)
self.recent_messages: List[Dict[str, str]] = []
self.recent_messages_limit = recent_messages_limit
self.max_summary_tokens = max_summary_tokens
self.summary: str = ""
def _summarize(self, messages_to_summarize: List[Dict[str, str]]) -> str:
"""Generate a summary of older messages."""
conversation = "\n".join(
[f"{m['role']}: {m['content']}" for m in messages_to_summarize]
)
prompt = (
f"Summarize the following conversation, preserving key facts, "
f"decisions, and user preferences. Be concise.\n\n{conversation}"
)
response = self.llm.complete(prompt)
return response.text
def put(self, key: str, value: Dict[str, Any]) -> None:
"""Add a message to memory."""
self.recent_messages.append(value)
# If we exceed the limit, summarize the oldest messages
if len(self.recent_messages) > self.recent_messages_limit:
messages_to_summarize = self.recent_messages[:2]
new_summary = self._summarize(messages_to_summarize)
if self.summary:
combined = f"Previous summary: {self.summary}\n\nNew information: {new_summary}"
self.summary = self._summarize([
{"role": "user", "content": combined}
])
else:
self.summary = new_summary
self.recent_messages = self.recent_messages[2:]
def get(self, key: str) -> List[ChatMessage]:
"""Retrieve the full memory as chat messages."""
messages = []
if self.summary:
messages.append(ChatMessage(
role=MessageRole.SYSTEM,
content=f"Conversation summary so far:\n{self.summary}"
))
for msg in self.recent_messages:
role = MessageRole.USER if msg["role"] == "user" else MessageRole.ASSISTANT
messages.append(ChatMessage(role=role, content=msg["content"]))
return messages
def get_all(self) -> Dict[str, Any]:
return {
"summary": self.summary,
"recent_messages": self.recent_messages,
}
def put_all(self, key: str, values: List[Dict[str, Any]]) -> None:
for value in values:
self.put(key, value)
def set(self, key: str, value: Any) -> None:
if isinstance(value, dict) and "summary" in value:
self.summary = value["summary"]
self.recent_messages = value.get("recent_messages", [])
@classmethod
def from_defaults(cls, **kwargs) -> "SummaryMemory":
return cls(**kwargs)
# Usage with an agent
summary_memory = SummaryMemory(
llm=OpenAI(model="gpt-4o-mini"),
recent_messages_limit=6,
)
agent = FunctionAgent(
tools=[],
llm=OpenAI(model="gpt-4o-mini"),
memory=summary_memory,
system_prompt="You are a helpful assistant with summary-based memory.",
)
import asyncio
async def demo():
topics = [
"My name is Alex and I love hiking.",
"I recently climbed Mount Rainier.",
"The trail was 14 miles round trip.",
"I want to plan a trip to the Grand Canyon next.",
"What gear do I need for desert hiking?",
"Remind me what mountains I've climbed recently.",
]
for topic in topics:
response = await agent.run(user_msg=topic)
print(f"User: {topic}")
print(f"Agent: {response}\n")
asyncio.run(demo())
This SummaryMemory class keeps the most recent six messages in full while compressing older history into a running summary. The agent can still recall that Alex climbed Mount Rainier even after that message has been summarized away.
Persisting Memory Across Sessions
For production agents, memory must survive restarts. LlamaIndex supports persistent chat stores and vector stores. Here's how to persist ChatMemoryBuffer using a key-value store:
from llama_index.core.storage.chat_store import SimpleChatStore
from llama_index.core.memory import ChatMemoryBuffer
from pathlib import Path
# Load existing store or create new one
store_path = Path("./chat_store.json")
if store_path.exists():
chat_store = SimpleChatStore.from_persist_path(str(store_path))
else:
chat_store = SimpleChatStore()
memory = ChatMemoryBuffer.from_defaults(
token_limit=4000,
chat_store=chat_store,
chat_store_key="user_sarah_session_1",
)
# Use the memory with an agent...
# After conversation, persist
chat_store.persist(persist_path=str(store_path))
print("Memory persisted to disk.")
On the next session, loading from chat_store.json restores the full conversation history. For multi-user systems, use distinct chat_store_key values per user or session.
Best Practices for Agent Memory
Choose the Right Memory Type for Your Use Case
Not every agent needs vector-backed long-term memory. A simple ChatMemoryBuffer is sufficient for most task-oriented conversations. Add long-term memory only when you need cross-session continuity or have large histories that exceed context windows.
Set Appropriate Token Limits
Token limits prevent memory from consuming your entire context window. A good rule of thumb is to allocate 30-50% of your context window to memory, leaving room for the system prompt, tools, and the response. Monitor actual usage and adjust.
Implement Memory Cleanup
Old or irrelevant memories accumulate over time. Implement cleanup strategies:
- Set TTL (time-to-live) on stored memories
- Periodically re-summarize and prune long-term storage
- Let users explicitly forget or correct stored facts
- Deduplicate semantically similar memories
Handle Memory Retrieval Failures Gracefully
Vector retrieval isn't perfect. Sometimes irrelevant memories surface, or the store is unavailable. Always have a fallback — the agent should function even if long-term memory is empty or returns poor results.
Log and Debug Memory Operations
Instrument your memory layer with logging so you can see what's being stored and retrieved. This is invaluable for debugging why an agent "forgot" something or retrieved irrelevant context:
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("agent_memory")
def recall_with_logging(query: str, top_k: int = 3):
logger.info(f"Recalling memories for: {query}")
retriever = memory_index.as_retriever(similarity_top_k=top_k)
results = retriever.retrieve(query)
for r in results:
logger.info(f" Retrieved (score={r.score:.3f}): {r.get_content()[:80]}")
return results
Consider Privacy and Data Retention
Memory systems store user data, which raises privacy concerns. Encrypt sensitive data at rest, comply with regulations like GDPR by supporting deletion requests, and avoid storing personally identifiable information in plain text in vector stores.
Test Memory Behavior Explicitly
Write tests that verify your agent remembers what it should and forgets what it should. For example:
import pytest
@pytest.mark.asyncio
async def test_agent_remembers_user_name(agent):
await agent.run(user_msg="My name is Test User.")
response = await agent.run(user_msg="What is my name?")
assert "Test User" in response
@pytest.mark.asyncio
async def test_agent_forgets_old_context(agent):
# Fill memory beyond the limit
for i in range(20):
await agent.run(user_msg=f"Random message number {i}")
response = await agent.run(user_msg="What was random message number 0?")
# The agent should NOT remember message 0 if it was trimmed
assert "random message number 0" not in response.lower()
Conclusion
Memory is what transforms a stateless LLM into a genuinely useful agent. LlamaIndex provides a rich set of memory primitives — from simple conversation buffers to composable memory blocks and custom architectures — that let you build agents matching your specific requirements. Start with ChatMemoryBuffer for short-term context, add vector-backed long-term memory when you need persistence, and reach for memory blocks or custom implementations when your use case demands sophisticated orchestration. By following the best practices around token limits, cleanup, privacy, and testing, you can build agents that remember what matters and forget what doesn't, delivering personalized and contextually aware experiences at scale.