Agent Memory Architectures with llama.cpp: Complete Guide
Building autonomous agents with large language models is one of the most exciting frontiers in software development today. However, an agent without memory is like a goldfish — it forgets everything the moment a new conversation begins. In this guide, we'll explore how to build robust agent memory architectures using llama.cpp, the popular C++ inference engine for running LLMs locally and efficiently.
What Is an Agent Memory Architecture?
An agent memory architecture is the system that allows an AI agent to store, retrieve, and use information across interactions. Just as humans rely on different types of memory — working memory for immediate tasks, episodic memory for past experiences, and semantic memory for general knowledge — AI agents benefit from a layered memory system that mirrors these cognitive functions.
With llama.cpp, you have full control over the inference pipeline, which means you can design memory systems that feed context into the model at exactly the right moments. Unlike API-based solutions where you're limited by request boundaries, a local llama.cpp setup lets you maintain persistent KV caches, manage context windows programmatically, and build custom retrieval pipelines.
Why Memory Matters for Agents
Without memory, every interaction with your agent starts from zero. This creates several critical problems:
- Context loss: The agent cannot reference prior decisions, user preferences, or ongoing tasks.
- Repetition: The agent may ask the same questions or make the same mistakes repeatedly.
- No learning: The agent cannot improve over time based on past experiences.
- Poor personalization: Every user interaction feels generic and disconnected.
- Limited reasoning: Complex multi-step tasks require holding intermediate results, which is impossible without working memory.
A well-designed memory architecture transforms a stateless text generator into a persistent, learning, context-aware agent capable of handling complex workflows.
Types of Agent Memory
Before diving into implementation, let's define the four primary types of memory you'll want to incorporate into your agent architecture.
Short-Term / Working Memory
Short-term memory holds the current conversation context — the messages exchanged in the active session. In llama.cpp, this maps directly to the context window. The model processes these tokens in real-time, and they form the immediate basis for generation.
Long-Term Memory
Long-term memory persists across sessions. This is where you store user preferences, learned facts, and historical interaction data. Typically implemented using vector databases or key-value stores, long-term memory requires a retrieval mechanism to surface relevant information when needed.
Episodic Memory
Episodic memory records specific past events or interactions — "what happened when." This allows an agent to recall that last Tuesday it helped a user debug a Python script, or that a particular approach failed in a previous attempt. Episodic memory is especially valuable for agents that perform recurring tasks.
Semantic Memory
Semantic memory stores general knowledge and facts that the agent has learned, independent of when or how they were acquired. This might include domain-specific information, user-provided documentation, or distilled insights from past conversations.
Setting Up llama.cpp for Agent Development
First, ensure you have llama.cpp built and a model downloaded. For this tutorial, we'll use the Python bindings (llama-cpp-python) for clarity, but the same concepts apply to the C++ API directly.
# Install llama-cpp-python
pip install llama-cpp-python
# Install supporting libraries
pip install numpy faiss-cpu sentence-transformersDownload a model in GGUF format. A good starting point is a quantized Llama or Mistral model:
# Example: download a quantized model
wget https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.ggufNow let's create a basic agent foundation:
from llama_cpp import Llama
import json
from datetime import datetime
# Initialize the model
llm = Llama(
model_path="mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_ctx=4096,
n_gpu_layers=-1, # Use all GPU layers if available
verbose=False
)
class Agent:
def __init__(self, llm, system_prompt="You are a helpful assistant."):
self.llm = llm
self.system_prompt = system_prompt
self.working_memory = []
def add_message(self, role, content):
self.working_memory.append({
"role": role,
"content": content,
"timestamp": datetime.now().isoformat()
})
def chat(self, user_input):
self.add_message("user", user_input)
messages = [{"role": "system", "content": self.system_prompt}]
messages.extend([
{"role": m["role"], "content": m["content"]}
for m in self.working_memory
])
response = self.llm.create_chat_completion(
messages=messages,
max_tokens=512,
temperature=0.7
)
reply = response["choices"][0]["message"]["content"]
self.add_message("assistant", reply)
return reply
# Basic usage
agent = Agent(llm)
print(agent.chat("Hello! My name is Alice."))
print(agent.chat("What's my name?")) # Works within this sessionThis basic agent has working memory — it remembers the conversation within a session. But once you restart, everything is lost. Let's fix that.
Building a Long-Term Memory Store
For long-term memory, we'll use a vector store with FAISS and sentence embeddings. This allows the agent to retrieve relevant past information based on semantic similarity.
import numpy as np
from sentence_transformers import SentenceTransformer
import faiss
import pickle
import os
class LongTermMemory:
def __init__(self, embedding_model_name="all-MiniLM-L6-v2",
storage_path="./memory_store"):
self.storage_path = storage_path
self.encoder = SentenceTransformer(embedding_model_name)
self.dimension = self.encoder.get_sentence_embedding_dimension()
self.index = faiss.IndexFlatIP(self.dimension)
self.memories = []
self._load()
def _load(self):
if os.path.exists(f"{self.storage_path}/memories.pkl"):
with open(f"{self.storage_path}/memories.pkl", "rb") as f:
self.memories = pickle.load(f)
if os.path.exists(f"{self.storage_path}/index.faiss"):
self.index = faiss.read_index(f"{self.storage_path}/index.faiss")
def _save(self):
os.makedirs(self.storage_path, exist_ok=True)
with open(f"{self.storage_path}/memories.pkl", "wb") as f:
pickle.dump(self.memories, f)
faiss.write_index(self.index, f"{self.storage_path}/index.faiss")
def store(self, text, metadata=None):
embedding = self.encoder.encode([text], normalize_embeddings=True)
memory_entry = {
"id": len(self.memories),
"text": text,
"metadata": metadata or {},
"timestamp": datetime.now().isoformat()
}
self.memories.append(memory_entry)
self.index.add(embedding.astype(np.float32))
self._save()
return memory_entry["id"]
def retrieve(self, query, top_k=5):
if len(self.memories) == 0:
return []
query_embedding = self.encoder.encode(
[query], normalize_embeddings=True
).astype(np.float32)
scores, indices = self.index.search(query_embedding, top_k)
results = []
for score, idx in zip(scores[0], indices[0]):
if idx >= 0 and idx < len(self.memories):
result = self.memories[idx].copy()
result["score"] = float(score)
results.append(result)
return resultsThis long-term memory store handles persistence, semantic search, and metadata. Now let's integrate it with our agent.
Integrating Memory Layers into the Agent
The key insight is that memory retrieval should happen before each generation. The agent queries its long-term memory for relevant context, then injects that context into the prompt alongside the working memory.
class MemoryAgent:
def __init__(self, llm, long_term_memory, system_prompt=None):
self.llm = llm
self.ltm = long_term_memory
self.working_memory = []
self.system_prompt = system_prompt or (
"You are a helpful, knowledgeable assistant with memory capabilities. "
"Use the provided context from past interactions to give personalized, "
"consistent responses. If context is provided, reference it naturally."
)
def _build_context_block(self, retrieved_memories):
if not retrieved_memories:
return ""
context_lines = ["[RELEVANT MEMORIES FROM PAST INTERACTIONS]"]
for mem in retrieved_memories:
ts = mem.get("timestamp", "unknown")[:10]
context_lines.append(f"- ({ts}) {mem['text']}")
context_lines.append("[END MEMORIES]\n")
return "\n".join(context_lines)
def _extract_memorable_info(self, user_input, response):
"""Use the LLM to decide what's worth remembering."""
extraction_prompt = (
"Analyze this interaction and extract any facts worth remembering "
"for future conversations (user preferences, personal info, decisions, "
"important context). Return a JSON array of strings, each being a "
"concise memory. Return [] if nothing is worth remembering.\n\n"
f"User: {user_input}\n"
f"Assistant: {response}\n\n"
"JSON array:"
)
result = self.llm(
extraction_prompt,
max_tokens=200,
temperature=0.1,
stop=["]"]
)
try:
text = result["choices"][0]["text"].strip()
if not text.startswith("["):
text = "[" + text
text += "]"
memories = json.loads(text)
return memories if isinstance(memories, list) else []
except (json.JSONDecodeError, KeyError):
return []
def chat(self, user_input):
# Retrieve relevant memories
retrieved = self.ltm.retrieve(user_input, top_k=5)
context_block = self._build_context_block(retrieved)
# Build the message list
self.working_memory.append({
"role": "user",
"content": user_input,
"timestamp": datetime.now().isoformat()
})
messages = [{"role": "system", "content": self.system_prompt}]
if context_block:
messages.append({
"role": "system",
"content": context_block
})
messages.extend([
{"role": m["role"], "content": m["content"]}
for m in self.working_memory[-20:] # Keep last 20 messages
])
response = self.llm.create_chat_completion(
messages=messages,
max_tokens=512,
temperature=0.7
)
reply = response["choices"][0]["message"]["content"]
self.working_memory.append({
"role": "assistant",
"content": reply,
"timestamp": datetime.now().isoformat()
})
# Extract and store new memories
new_memories = self._extract_memorable_info(user_input, reply)
for mem in new_memories:
self.ltm.store(mem, metadata={
"user_input": user_input,
"response": reply
})
return replyNow let's see it in action:
# Initialize components
ltm = LongTermMemory(storage_path="./agent_memory")
agent = MemoryAgent(llm, ltm)
# Session 1
print(agent.chat("Hi! I'm Alice and I prefer Python for data science work."))
print(agent.chat("I'm working on a machine learning project about weather prediction."))
# Session 2 (even after restarting the program)
agent2 = MemoryAgent(llm, ltm) # Same storage path
print(agent2.chat("What programming language do I prefer?"))
# Agent recalls: Python, from past interaction
print(agent2.chat("What project am I working on?"))
# Agent recalls: weather prediction ML projectAdvanced: Episodic Memory with Summarization
For episodic memory, we want to store summaries of entire conversation episodes. This is more efficient than storing every message and allows the agent to recall high-level patterns from past sessions.
class EpisodicMemory:
def __init__(self, llm, long_term_memory):
self.llm = llm
self.ltm = long_term_memory
self.current_episode = []
def add_interaction(self, user_input, response):
self.current_episode.append({
"user": user_input,
"assistant": response
})
def summarize_and_store(self):
if not self.current_episode:
return None
transcript = "\n".join([
f"User: {i['user']}\nAssistant: {i['assistant']}"
for i in self.current_episode
])
summary_prompt = (
"Summarize this conversation episode in 2-3 sentences, "
"capturing key topics, decisions, and any user preferences "
"or important facts revealed:\n\n"
f"{transcript}\n\nSummary:"
)
result = self.llm(
summary_prompt,
max_tokens=150,
temperature=0.3
)
summary = result["choices"][0]["text"].strip()
memory_id = self.ltm.store(
summary,
metadata={
"type": "episodic",
"interaction_count": len(self.current_episode),
"timestamp": datetime.now().isoformat()
}
)
self.current_episode = []
return memory_idIntegrate episodic memory into your agent by calling summarize_and_store() at the end of each session or when the working memory reaches a threshold.
Managing Context Window Limits
One of the most important practical considerations with llama.cpp is the context window limit. Even with a 4096 or 8192 token context, you can quickly run out of space when injecting retrieved memories, conversation history, and system prompts. Here's a strategy for managing this:
class ContextManager:
def __init__(self, max_context_tokens=3500, reserve_for_response=512):
self.max_context = max_context_tokens
self.reserve = reserve_for_response
def estimate_tokens(self, text):
"""Rough estimate: ~4 characters per token."""
return len(text) // 4
def build_optimized_context(self, system_prompt, retrieved_memories,
working_memory, user_input):
budget = self.max_context - self.reserve
messages = []
# Always include system prompt
system_tokens = self.estimate_tokens(system_prompt)
budget -= system_tokens
messages.append({"role": "system", "content": system_prompt})
# Add retrieved memories (highest priority context)
if retrieved_memories:
memory_text = "\n".join([
f"- {m['text']}" for m in retrieved_memories[:3]
])
memory_block = f"[Past memories]:\n{memory_text}"
memory_tokens = self.estimate_tokens(memory_block)
if memory_tokens < budget:
messages.append({"role": "system", "content": memory_block})
budget -= memory_tokens
# Add working memory from oldest to newest,
# stopping when budget is exhausted
for msg in reversed(working_memory):
msg_tokens = self.estimate_tokens(msg["content"])
if msg_tokens > budget:
break
messages.insert(-1 if retrieved_memories else 1, {
"role": msg["role"],
"content": msg["content"]
})
budget -= msg_tokens
return messagesThis context manager prioritizes system prompts, then retrieved memories, then the most recent conversation history — dropping older messages first when space runs out.
Best Practices for Agent Memory Architectures
1. Be Selective About What You Store
Not every interaction is worth remembering. Use an LLM-based extraction step (as shown above) to filter out small talk and retain only meaningful information. This keeps your memory store clean and retrieval accurate.
2. Implement Memory Decay
Old memories may become irrelevant. Implement a decay mechanism that reduces the retrieval score of older entries or periodically prunes low-value memories:
def retrieve_with_decay(self, query, top_k=5, decay_factor=0.99):
results = self.retrieve(query, top_k=top_k * 2)
now = datetime.now()
for result in results:
age_days = (now - datetime.fromisoformat(result["timestamp"])).days
result["score"] *= (decay_factor ** age_days)
results.sort(key=lambda x: x["score"], reverse=True)
return results[:top_k]3. Use Structured Memory When Possible
For facts like user preferences, project details, or configuration, use structured storage (JSON, SQLite) alongside vector search. This allows exact lookups without relying on semantic similarity:
import sqlite3
class StructuredMemory:
def __init__(self, db_path="agent_facts.db"):
self.conn = sqlite3.connect(db_path)
self.conn.execute("""
CREATE TABLE IF NOT EXISTS facts (
key TEXT PRIMARY KEY,
value TEXT,
category TEXT,
updated_at TEXT
)
""")
self.conn.commit()
def set_fact(self, key, value, category="general"):
self.conn.execute(
"INSERT OR REPLACE INTO facts VALUES (?, ?, ?, ?)",
(key, value, category, datetime.now().isoformat())
)
self.conn.commit()
def get_fact(self, key):
cursor = self.conn.execute(
"SELECT value FROM facts WHERE key = ?", (key,)
)
row = cursor.fetchone()
return row[0] if row else None
def get_all_facts(self, category=None):
if category:
cursor = self.conn.execute(
"SELECT key, value FROM facts WHERE category = ?", (category,)
)
else:
cursor = self.conn.execute("SELECT key, value FROM facts")
return {row[0]: row[1] for row in cursor.fetchall()}4. Handle Memory Conflicts
Users may contradict earlier statements ("Actually, I now prefer Rust over Python"). Your agent should detect updates and overwrite or annotate old memories rather than storing conflicting versions.
5. Log and Debug Memory Operations
Memory systems can produce surprising behavior. Always log what's being stored and retrieved so you can debug issues:
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("agent_memory")
# In your retrieve method:
logger.info(f"Retrieving memories for query: {query}")
logger.info(f"Found {len(results)} results, top score: {results[0]['score']:.3f}")
# In your store method:
logger.info(f"Storing memory: {text[:80]}...")6. Leverage llama.cpp KV Cache Reuse
llama.cpp supports KV cache reuse, which can significantly speed up multi-turn conversations. When the prefix of your prompt hasn't changed, the model can skip reprocessing those tokens. Structure your prompts so that stable content (system prompt, early conversation) comes first, and volatile content (newest messages) comes last.
7. Test Memory Retrieval Quality
Build a test suite that verifies your agent recalls the right information:
def test_memory_recall():
ltm = LongTermMemory(storage_path="./test_memory")
agent = MemoryAgent(llm, ltm)
# Store known facts
agent.chat("My favorite database is PostgreSQL.")
agent.chat("I deploy my apps on AWS.")
# Test recall
response = agent.chat("What database do I like?")
assert "postgresql" in response.lower() or "postgres" in response.lower()
response = agent.chat("Where do I deploy?")
assert "aws" in response.lower()
print("All memory tests passed!")
test_memory_recall()Putting It All Together
Here's a complete example that combines all memory types into a single cohesive agent:
class FullMemoryAgent:
def __init__(self, llm, storage_dir="./agent_data"):
self.llm = llm
self.ltm = LongTermMemory(storage_path=f"{storage_dir}/ltm")
self.structured = StructuredMemory(db_path=f"{storage_dir}/facts.db")
self.episodic = EpisodicMemory(llm, self.ltm)
self.context_mgr = ContextManager(max_context_tokens=3500)
self.working_memory = []
self.system_prompt = (
"You are a helpful assistant with persistent memory. "
"You can recall past interactions and user preferences. "
"Always use provided context naturally. "
"If you learn new facts about the user, mention them "
"in your response so they know you remember."
)
def chat(self, user_input):
# Retrieve from long-term memory
ltm_results = self.ltm.retrieve(user_input, top_k=3)
# Get relevant structured facts
all_facts = self.structured.get_all_facts()
facts_str = json.dumps(all_facts, indent=2) if all_facts else ""
# Build context
context_parts = []
if ltm_results:
context_parts.append("Past memories:\n" +
"\n".join([f"- {m['text']}" for m in ltm_results]))
if facts_str:
context_parts.append(f"Known facts:\n{facts_str}")
context = "\n\n".join(context_parts)
# Add to working memory
self.working_memory.append({
"role": "user",
"content": user_input
})
# Build messages with context management
messages = [{"role": "system", "content": self.system_prompt}]
if context:
messages.append({"role": "system", "content": context})
messages.extend(self.working_memory[-15:])
# Generate response
response = self.llm.create_chat_completion(
messages=messages,
max_tokens=512,
temperature=0.7
)
reply = response["choices"][0]["message"]["content"]
self.working_memory.append({
"role": "assistant",
"content": reply
})
# Record for episodic memory
self.episodic.add_interaction(user_input, reply)
# Extract and store new memories
new_memories = self._extract_memorable_info(user_input, reply)
for mem in new_memories:
self.ltm.store(mem)
return reply
def end_session(self):
"""Call this when a conversation session ends."""
self.episodic.summarize_and_store()
self.working_memory = []
def _extract_memorable_info(self, user_input, response):
prompt = (
"Extract memorable facts from this exchange as a JSON array "
"of {key, value, category} objects. Categories: preference, "
"project, personal, technical, other. Return [] if nothing "
"notable.\n\n"
f"User: {user_input}\nAssistant: {response}\n\nJSON:"
)
result = self.llm(prompt, max_tokens=200, temperature=0.1)
try:
text = result["choices"][0]["text"].strip()
items = json.loads(text if text.startswith("[") else "[" + text + "]")
return items if isinstance(items, list) else []
except (json.JSONDecodeError, KeyError):
return []
# Usage
agent = FullMemoryAgent(llm)
print(agent.chat("I'm Bob, a backend engineer who loves Go and Kubernetes."))
print(agent.chat("I'm building a microservice for payment processing."))
print(agent.chat("What do you know about me?"))
agent.end_session() # Summarizes and stores the episodeConclusion
Building effective agent memory architectures with llama.cpp is about layering different memory types — working, long-term, episodic, and semantic — and intelligently managing what gets stored, when it gets retrieved, and how it fits within the model's context window. By combining vector search for semantic recall, structured storage for exact facts, LLM-based extraction for filtering, and careful context management, you can create agents that genuinely learn from their interactions and provide increasingly personalized, context-aware responses over time. The local inference capabilities of llama.cpp make this especially powerful, as you have full control over the pipeline, can run everything privately, and can optimize performance through KV cache reuse and custom prompt engineering. Start simple with working memory and a basic vector store, then gradually add episodic summarization, structured facts, and decay mechanisms as your agent's needs grow.