How to Implement Stateful Conversations with Local Models

How to Implement Stateful Conversations with Local Models

Stateful conversations are the backbone of any meaningful chatbot or AI assistant experience. Unlike one-off completions, where each prompt is treated independently, stateful conversations retain context across multiple turns — allowing the model to reference earlier messages, remember user preferences, and maintain coherent dialogue. When you combine this with local models (models running on your own hardware via tools like Ollama, llama.cpp, or Hugging Face Transformers), you gain full control over data privacy, latency, and cost.

This tutorial walks through everything you need to know: what stateful conversations are, why they matter, how to implement them with local models, and the best practices that separate toy demos from production-ready systems.

What Is a Stateful Conversation?

A stateful conversation is one where the system maintains a record of the exchange between the user and the assistant. Each new user message is processed in the context of the entire prior conversation, not in isolation. The "state" is simply the running history of messages — typically stored as a list of role-tagged entries (user, assistant, and sometimes system).

Local models are large language models that run entirely on your own machine or servers. Popular options include Llama 3, Mistral, Phi-3, and Gemma, served through runtimes like Ollama or llama.cpp. Because these models are stateless by design — they accept a prompt and return a completion — the responsibility for managing conversation state falls on your application code.

Why Stateful Conversations Matter

• Contextual awareness: The model can reference earlier statements, answer follow-up questions, and avoid repeating itself.
• Privacy and control: With local models, conversation history never leaves your infrastructure, which is critical for healthcare, legal, and enterprise use cases.
• Cost efficiency: You avoid per-token API charges. Once the model is loaded, inference is essentially free.
• Latency: Local inference eliminates network round-trips, enabling faster interactive experiences.
• Customization: You can fine-tune the model, adjust system prompts per session, and inject domain-specific context freely.

How to Implement Stateful Conversations

The core implementation pattern is straightforward: maintain a list of message dictionaries, append each new user message, send the entire list to the model, append the model's response, and repeat. Let's build this step by step using Ollama, one of the most popular local model runtimes.

Prerequisites

Before you begin, install Ollama and pull a model. On macOS or Linux, you can install Ollama from the official website, then run:

ollama pull llama3.2
ollama serve

This starts a local API server at http://localhost:11434. You will also need Python 3.9+ and the requests library:

pip install requests

Step 1: Define the Conversation State Class

The cleanest approach is to encapsulate conversation state in a class. This keeps the message history, system prompt, and model configuration together in one manageable object.

import requests
import json

class StatefulConversation:
    def __init__(self, model="llama3.2", system_prompt="You are a helpful assistant."):
        self.model = model
        self.messages = [{"role": "system", "content": system_prompt}]
        self.api_url = "http://localhost:11434/api/chat"

    def add_user_message(self, content):
        self.messages.append({"role": "user", "content": content})

    def add_assistant_message(self, content):
        self.messages.append({"role": "assistant", "content": content})

    def send(self, user_input):
        self.add_user_message(user_input)
        payload = {
            "model": self.model,
            "messages": self.messages,
            "stream": False
        }
        response = requests.post(self.api_url, json=payload)
        response.raise_for_status()
        assistant_content = response.json()["message"]["content"]
        self.add_assistant_message(assistant_content)
        return assistant_content

    def get_history(self):
        return self.messages

Notice how the send method appends the user's message, sends the entire message history to the model, and then appends the assistant's response. This is the fundamental loop of stateful conversation.

Step 2: Run an Interactive Conversation

Now let's use the class in a simple REPL-style loop:

if __name__ == "__main__":
    convo = StatefulConversation(
        model="llama3.2",
        system_prompt="You are a friendly coding tutor. Answer concisely."
    )

    print("Chat started. Type 'quit' to exit.\n")

    while True:
        user_input = input("You: ")
        if user_input.strip().lower() in ("quit", "exit"):
            break
        reply = convo.send(user_input)
        print(f"Assistant: {reply}\n")

    print("\n--- Conversation History ---")
    for msg in convo.get_history():
        print(f"[{msg['role']}] {msg['content']}")

When you run this, you can ask follow-up questions and the model will remember the context. For example:

You: My name is Alice.
Assistant: Nice to meet you, Alice! How can I help you today?

You: What did I just tell you my name was?
Assistant: You told me your name is Alice.

Without state management, the second question would fail because the model would have no memory of the first exchange.

Step 3: Persisting Conversations Across Sessions

In-memory state is lost when your application restarts. For real applications, you need to persist conversation history to disk or a database. Here is a simple JSON-based persistence layer:

import os

class PersistentConversation(StatefulConversation):
    def __init__(self, session_id, model="llama3.2", system_prompt="You are a helpful assistant.", storage_dir="sessions"):
        self.session_id = session_id
        self.storage_dir = storage_dir
        os.makedirs(storage_dir, exist_ok=True)
        self.file_path = os.path.join(storage_dir, f"{session_id}.json")

        if os.path.exists(self.file_path):
            self.messages = self._load()
            self.model = model
            self.api_url = "http://localhost:11434/api/chat"
        else:
            super().__init__(model, system_prompt)
            self._save()

    def _load(self):
        with open(self.file_path, "r", encoding="utf-8") as f:
            return json.load(f)

    def _save(self):
        with open(self.file_path, "w", encoding="utf-8") as f:
            json.dump(self.messages, f, indent=2)

    def send(self, user_input):
        reply = super().send(user_input)
        self._save()
        return reply

Now each conversation session is identified by a unique ID and saved to disk after every turn. You can resume any session by constructing a PersistentConversation with the same session_id.

Step 4: Managing Context Window Limits

Every model has a maximum context window — the total number of tokens it can process at once. For Llama 3.2, this is typically 128,000 tokens, but smaller models may have 4,000 or 8,000. As conversations grow longer, you will eventually exceed this limit. You need a strategy to trim or summarize old messages.

A common approach is a sliding window that keeps the system prompt and the most recent N messages:

class ManagedConversation(StatefulConversation):
    def __init__(self, model="llama3.2", system_prompt="You are a helpful assistant.", max_messages=20):
        super().__init__(model, system_prompt)
        self.max_messages = max_messages

    def _trim_history(self):
        if len(self.messages) <= self.max_messages:
            return
        system_messages = [m for m in self.messages if m["role"] == "system"]
        non_system = [m for m in self.messages if m["role"] != "system"]
        trimmed = non_system[-(self.max_messages - len(system_messages)):]
        self.messages = system_messages + trimmed

    def send(self, user_input):
        self._trim_history()
        return super().send(user_input)

For more sophisticated use cases, you can implement summarization: periodically send the oldest messages to the model with a prompt like "Summarize this conversation so far," then replace those messages with a single system message containing the summary.

Step 5: Multi-Session Management

Real applications often handle many concurrent users, each with their own conversation state. A session manager keeps track of active conversations by ID:

class SessionManager:
    def __init__(self, model="llama3.2", default_system_prompt="You are a helpful assistant."):
        self.model = model
        self.default_system_prompt = default_system_prompt
        self.sessions = {}

    def get_or_create(self, session_id):
        if session_id not in self.sessions:
            self.sessions[session_id] = PersistentConversation(
                session_id=session_id,
                model=self.model,
                system_prompt=self.default_system_prompt
            )
        return self.sessions[session_id]

    def send(self, session_id, user_input):
        convo = self.get_or_create(session_id)
        return convo.send(user_input)

    def list_sessions(self):
        return list(self.sessions.keys())

You can then expose this through a web framework like FastAPI:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
manager = SessionManager(model="llama3.2")

class ChatRequest(BaseModel):
    session_id: str
    message: str

@app.post("/chat")
def chat(req: ChatRequest):
    reply = manager.send(req.session_id, req.message)
    return {"reply": reply, "session_id": req.session_id}

Best Practices

• Always include a system prompt: The system message sets the model's behavior and persona. Keep it as the first entry in the message list and never trim it.
• Validate and sanitize input: Even with local models, user input should be sanitized to prevent prompt injection attacks that could override your system prompt.
• Handle errors gracefully: Local model servers can crash, run out of memory, or time out. Wrap inference calls in try/except blocks and provide fallback responses.
• Log conversations for debugging: Store transcripts (with user consent) so you can diagnose bad responses and improve your system prompts.
• Use streaming for better UX: Ollama supports streaming responses. For interactive applications, stream tokens to the client so users see output immediately rather than waiting for the full response.
• Choose the right model size: Larger models produce better responses but consume more memory and run slower. Match the model to your hardware and latency requirements.
• Experiment with temperature: Lower temperature values (0.1–0.3) produce more deterministic, factual responses. Higher values (0.7–1.0) produce more creative output. Adjust based on your use case.
• Consider RAG for long-term memory: For conversations that need to reference documents or knowledge beyond the context window, combine stateful chat with retrieval-augmented generation using a local vector database like ChromaDB.

Streaming Responses

For a more responsive experience, enable streaming so tokens are delivered as they are generated. Here is how to modify the send method for streaming:

def send_streaming(self, user_input):
    self.add_user_message(user_input)
    payload = {
        "model": self.model,
        "messages": self.messages,
        "stream": True
    }
    response = requests.post(self.api_url, json=payload, stream=True)
    response.raise_for_status()

    full_response = ""
    for line in response.iter_lines():
        if line:
            chunk = json.loads(line)
            token = chunk["message"]["content"]
            full_response += token
            print(token, end="", flush=True)

    print()  # newline after streaming completes
    self.add_assistant_message(full_response)
    return full_response

Conclusion

Implementing stateful conversations with local models is a powerful pattern that gives you full ownership of your AI application's data, behavior, and costs. The core concept is simple — maintain a running list of messages and send the full history with each request — but production-quality implementations require attention to persistence, context window management, multi-session handling, and error resilience. By following the architecture and best practices outlined in this tutorial, you can build chat experiences that rival cloud-based APIs while keeping every token of conversation data on hardware you control. Start with the basic StatefulConversation class, add persistence when you need session continuity, introduce context trimming as conversations grow, and layer in streaming and web APIs to serve real users. The result is a flexible, private, and cost-effective conversational AI system that you fully understand and can extend in any direction your application demands.