Context Window Management for Long-Horizon Agents

Context Window Management for Long-Horizon Agents

Long-horizon agents — systems that operate over many steps, hours, or even days — face a fundamental constraint: the context window of the underlying language model is finite. Whether you're building a coding agent that refactors an entire repository, a research assistant that browses dozens of sources, or a customer-support bot that maintains a multi-turn conversation, the agent will eventually accumulate more context than the model can ingest in a single call. Context window management is the discipline of deciding what stays, what gets compressed, what gets evicted, and what gets retrieved on demand so the agent remains coherent and effective across long horizons.

What Is Context Window Management?

At its core, context window management is the set of strategies and mechanisms an agent uses to keep its prompt within the model's token limit while preserving the information necessary to complete its task. A modern agent's context typically contains several categories of content:

• System prompt and instructions — the agent's role, tools, and policies.
• Conversation history — prior user messages and assistant responses.
• Tool calls and observations — results from code execution, API calls, or web fetches.
• Retrieved documents — chunks pulled from a vector store or search index.
• Scratchpad or working memory — intermediate reasoning, plans, and summaries.

Each of these grows at different rates. Tool observations, in particular, can explode in size — a single ls -R on a large repo or a verbose API response can consume thousands of tokens in one step. Without active management, the agent hits the token ceiling, degrades in reasoning quality, or fails outright with a context-length error.

Why It Matters

The naive approach — "just include everything" — fails for three reasons. First, there is a hard limit: every model has a maximum context length, and exceeding it produces an error or silent truncation. Second, even within the limit, models suffer from attention dilution: as context grows, the model's ability to focus on the most relevant information degrades, a phenomenon related to the "lost in the middle" effect. Third, cost and latency scale with input tokens. An agent that re-sends a 100K-token transcript on every step of a 200-step task is burning orders of magnitude more compute and money than necessary.

Effective context management directly improves three metrics that matter to developers: task success rate (the agent retains the right information to act correctly), cost per task (fewer redundant tokens), and latency (smaller prompts process faster). For production agents, these are often the difference between a viable product and an unsustainable one.

Core Strategies

There is no single technique that solves context management. Production systems combine several complementary strategies, each addressing a different failure mode.

1. Summarization and Compaction

The most common technique is to periodically summarize older portions of the context into a compact representation. When the context approaches a threshold, the agent (or a separate "compactor" call) takes the oldest N messages, produces a summary, and replaces them with that summary plus the most recent messages.

import tiktoken

def count_tokens(messages, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for m in messages:
        total += len(enc.encode(m["content"]))
    return total

def compact_history(messages, max_tokens, summarizer, keep_recent=6):
    """
    If total tokens exceed max_tokens, summarize everything except
    the most recent `keep_recent` messages into a single summary block.
    """
    if count_tokens(messages) <= max_tokens:
        return messages

    to_summarize = messages[:-keep_recent]
    recent = messages[-keep_recent:]

    summary_text = summarizer(to_summarize)

    return [
        {"role": "system", "content": f"Summary of earlier conversation:\n{summary_text}"},
        *recent,
    ]

The summarizer can be a smaller, cheaper model since summarization is a well-understood task. The key design decision is keep_recent: too few and you lose actionable detail; too many and compaction never triggers. A common heuristic is to keep enough recent turns to cover the current sub-task plus one or two prior steps for continuity.

2. Hierarchical Summarization

For very long horizons, a single flat summary becomes lossy. Hierarchical summarization maintains summaries at multiple levels of granularity: a high-level summary of the entire task, mid-level summaries of each phase, and the raw recent context. When the agent needs detail about a past phase, it can expand that mid-level summary back into its fuller form.

class HierarchicalMemory:
    def __init__(self):
        self.global_summary = ""
        self.phase_summaries = []   # list of (phase_id, summary, raw_messages)
        self.recent = []            # current phase raw messages

    def add_message(self, msg, compactor, phase_token_limit=4000):
        self.recent.append(msg)
        if count_tokens(self.recent) > phase_token_limit:
            summary = compactor(self.recent)
            self.phase_summaries.append({
                "phase_id": len(self.phase_summaries),
                "summary": summary,
                "raw": self.recent,
            })
            self.recent = []
            # Rebuild global summary from phase summaries
            self.global_summary = compactor(
                [{"role": "system", "content": p["summary"]}
                 for p in self.phase_summaries]
            )

    def build_prompt(self):
        parts = []
        if self.global_summary:
            parts.append({"role": "system",
                          "content": f"Task summary so far: {self.global_summary}"})
        for p in self.phase_summaries[-3:]:  # last 3 phase summaries
            parts.append({"role": "system",
                          "content": f"Phase {p['phase_id']}: {p['summary']}"})
        parts.extend(self.recent)
        return parts

This approach mirrors how humans handle long projects: you remember the overall arc, you remember what each major phase accomplished, and you keep detailed notes only for what you're actively working on.

3. Selective Retention and Eviction

Not all context is equally valuable. A tool call that returned a 5,000-token stack trace may be critical for the next step but irrelevant ten steps later. Selective retention applies explicit policies about what to keep based on message type and age.

def selective_prune(messages, max_tokens=8000):
    """
    Prune low-value content while preserving high-value items.
    """
    pruned = []
    budget = max_tokens

    # First pass: mark items for retention based on type and recency
    for i, msg in enumerate(reversed(messages)):
        age = i  # 0 = most recent
        content = msg.get("content", "")

        # Always keep the most recent 4 messages
        if age < 4:
            pruned.insert(0, msg)
            budget -= count_tokens([msg])
            continue

        # Keep tool results only if recent or explicitly marked important
        if msg.get("role") == "tool" and age > 8:
            pruned.insert(0, {"role": "tool",
                              "content": "[tool output evicted]",
                              "tool_call_id": msg.get("tool_call_id")})
            continue

        # Truncate very long messages to a snippet
        msg_tokens = count_tokens([msg])
        if msg_tokens > 1000:
            snippet = content[:1500] + "\n...[truncated]..."
            new_msg = {**msg, "content": snippet}
            pruned.insert(0, new_msg)
            budget -= count_tokens([new_msg])
        else:
            pruned.insert(0, msg)
            budget -= msg_tokens

        if budget <= 0:
            break

    return pruned

Notice the placeholder for evicted tool outputs. This preserves the structural integrity of the conversation (the assistant's message references a tool call ID, so the corresponding tool response must exist) while discarding the bulky content. Some frameworks also support "soft eviction," where the full content is moved to an external store and a reference is left in context.

4. External Memory with Retrieval

For agents that operate over hours or days, the most scalable approach is to offload context to an external store — a vector database, a key-value store, or even a simple file — and retrieve only what's needed for the current step. This decouples the agent's effective memory from the model's context window entirely.

import json
from pathlib import Path

class ExternalMemory:
    def __init__(self, store_path="agent_memory.jsonl"):
        self.store_path = Path(store_path)
        self.store_path.touch(exist_ok=True)

    def write(self, entry):
        """Append an entry with metadata for later retrieval."""
        with open(self.store_path, "a") as f:
            f.write(json.dumps(entry) + "\n")

    def search(self, query, embed_fn, top_k=5):
        """Simple keyword/semantic search over stored entries."""
        results = []
        with open(self.store_path) as f:
            for line in f:
                entry = json.loads(line)
                score = self._relevance(query, entry, embed_fn)
                results.append((score, entry))
        results.sort(key=lambda x: -x[0])
        return [e for _, e in results[:top_k]]

    def _relevance(self, query, entry, embed_fn):
        # In production, use cosine similarity over embeddings
        query_terms = set(query.lower().split())
        entry_terms = set(entry["content"].lower().split())
        return len(query_terms & entry_terms)

# Usage in an agent loop
memory = ExternalMemory()

def agent_step(user_input, llm, embed_fn):
    # Retrieve relevant past context
    relevant = memory.search(user_input, embed_fn, top_k=3)
    context_block = "\n---\n".join(r["content"] for r in relevant)

    prompt = [
        {"role": "system", "content": "You are a helpful agent."},
        {"role": "system", "content": f"Relevant past context:\n{context_block}"},
        {"role": "user", "content": user_input},
    ]

    response = llm(prompt)

    # Store this interaction for future retrieval
    memory.write({
        "content": f"User: {user_input}\nAssistant: {response}",
        "timestamp": "2024-01-01T00:00:00Z",
        "tags": ["conversation"],
    })

    return response

The retrieval approach shines when the agent's task spans many independent sub-tasks. A coding agent working through a backlog of issues doesn't need every prior issue's details in context — it needs the current issue, the relevant files, and perhaps a summary of related prior work. Retrieval makes that possible.

5. Structured Working Memory

Rather than dumping raw conversation history into the prompt, many advanced agents maintain a structured working-memory document that the agent itself updates. This might be a TODO list, a plan, a set of key facts, or a running log of decisions. The agent reads and writes this document each step, and only the document (not the full history) goes into the prompt.

WORKING_MEMORY_TEMPLATE = """# Current Task
{task}

# Plan
{plan}

# Key Facts Discovered
{facts}

# Completed Steps
{completed}

# Current Step
{current_step}

# Open Questions
{questions}
"""

class WorkingMemory:
    def __init__(self, task):
        self.task = task
        self.plan = []
        self.facts = []
        self.completed = []
        self.current_step = ""
        self.questions = []

    def render(self):
        return WORKING_MEMORY_TEMPLATE.format(
            task=self.task,
            plan="\n".join(f"- {p}" for p in self.plan),
            facts="\n".join(f"- {f}" for f in self.facts),
            completed="\n".join(f"- [x] {c}" for c in self.completed),
            current_step=self.current_step,
            questions="\n".join(f"- {q}" for q in self.questions),
        )

    def update_from_response(self, llm_response):
        """Parse the agent's output to update working memory fields."""
        # In practice, use structured output / function calling
        if "FACT:" in llm_response:
            for line in llm_response.split("\n"):
                if line.startswith("FACT:"):
                    self.facts.append(line[5:].strip())
        if "DONE:" in llm_response:
            for line in llm_response.split("\n"):
                if line.startswith("DONE:"):
                    self.completed.append(line[5:].strip())

This pattern is powerful because the agent controls its own memory footprint. It decides what's worth recording as a fact, what to mark complete, and what to keep as an open question. The prompt stays small and focused regardless of how many steps the agent has taken.

Putting It Together: A Complete Agent Loop

A production long-horizon agent typically combines several of these techniques. Here's a simplified but complete agent loop that uses working memory, selective pruning, and periodic compaction together:

class LongHorizonAgent:
    def __init__(self, llm, summarizer, max_context_tokens=12000):
        self.llm = llm
        self.summarizer = summarizer
        self.max_context_tokens = max_context_tokens
        self.working_memory = None
        self.history = []
        self.summary = ""

    def run(self, task, max_steps=50):
        self.working_memory = WorkingMemory(task)
        self.working_memory.plan = self._make_initial_plan(task)

        for step in range(max_steps):
            # Build prompt from working memory + recent history
            prompt = self._build_prompt()

            # Check context budget before calling the model
            if count_tokens(prompt) > self.max_context_tokens:
                self._compact()

            response = self.llm(prompt)
            self.history.append({"role": "assistant", "content": response})

            # Update working memory from the response
            self.working_memory.update_from_response(response)

            # Check for task completion
            if "TASK_COMPLETE" in response:
                return response

        return "Max steps reached without completion."

    def _build_prompt(self):
        prompt = [
            {"role": "system",
             "content": "You are a long-horizon task agent. "
                        "Use the working memory below. "
                        "Output TASK_COMPLETE when done."},
            {"role": "system",
             "content": self.working_memory.render()},
        ]

        if self.summary:
            prompt.append({"role": "system",
                           "content": f"Prior context summary: {self.summary}"})

        # Include only the last few raw exchanges for continuity
        recent = self.history[-4:]
        prompt.extend(recent)
        return prompt

    def _compact(self):
        """Summarize older history to free context budget."""
        if len(self.history) <= 4:
            return
        older = self.history[:-4]
        new_summary = self.summarizer(older)
        self.summary = f"{self.summary}\n{new_summary}".strip()
        self.history = self.history[-4:]

    def _make_initial_plan(self, task):
        plan_response = self.llm([
            {"role": "system", "content": "Break this task into steps."},
            {"role": "user", "content": task},
        ])
        return [line.strip("- ") for line in plan_response.split("\n")
                if line.strip().startswith("-")]

This architecture keeps the prompt bounded regardless of how many steps the agent takes. The working memory provides structured, agent-controlled context; the summary preserves the arc of the conversation; and the recent history gives the model enough raw detail to maintain coherence in the current step.

Best Practices

• Measure token usage at every step. Instrument your agent to log prompt size, completion size, and cost per step. You cannot manage what you do not measure, and token usage patterns often reveal surprising growth (e.g., a tool that returns progressively larger results).
• Set compaction thresholds conservatively. Trigger compaction at 60–70% of the context limit, not 95%. This leaves headroom for the model's response and for any tool outputs that arrive before the next compaction check.
• Preserve conversation structure. When pruning, never remove a tool-call message without also handling its corresponding tool-response. Most APIs require these to remain paired. Use placeholder content rather than deletion when you need to shrink tool outputs.
• Use cheaper models for summarization. Summarization is a well-suited task for smaller, faster models. Reserve your most capable (and expensive) model for the reasoning steps that actually advance the task.
• Make summaries queryable. If you evict content to an external store, ensure the agent can retrieve it when needed. An agent that has "forgotten" a critical detail with no way to recover it will fail in confusing ways.
• Test with long horizons explicitly. Many agent bugs only emerge after 30+ steps. Create test scenarios that run long enough to trigger compaction, eviction, and retrieval paths. A test that always completes in 5 steps will never exercise your context management code.
• Let the agent manage its own memory when possible. Structured working memory that the agent updates via tool calls or structured output tends to be more robust than heuristic pruning rules, because the agent can reason about what it needs.
• Watch for information loss cascades. If a summary drops a key fact, every subsequent summary built on top of it also lacks that fact. Periodically verify that critical information survives compaction by including key-fact checkpoints in your working memory.

Conclusion

Context window management is not an optimization you bolt on after building an agent — it is a core architectural concern that shapes how the agent is designed from the first step. The most robust long-horizon agents combine structured working memory for agent-controlled focus, summarization and hierarchical compaction for preserving the task arc, selective eviction for discarding low-value content, and external retrieval for accessing the full history on demand. By treating context as a managed resource rather than an unlimited buffer, you build agents that remain coherent, cost-effective, and capable across tasks that span hundreds of steps — the kind of tasks where agents deliver their greatest value.