How to Implement Streaming Responses with Local Models

Streaming responses have become a defining feature of modern AI applications. Instead of waiting for an entire response to be generated before displaying anything, streaming sends tokens to the client as soon as they are produced. When working with local models — running on your own hardware through tools like Ollama, llama.cpp, vLLM, or Hugging Face Transformers — streaming is not only possible but often easier to implement than with cloud APIs, because you have direct access to the inference loop.

This tutorial walks through what streaming responses are, why they matter for local model deployments, how to implement them across several popular local inference stacks, and the best practices that will keep your implementation fast, robust, and user-friendly.

What Is a Streaming Response?

A streaming response is a way of delivering model output incrementally. Rather than the server computing the full completion and returning one large JSON payload, it emits a sequence of small chunks — usually one token (or a few tokens) at a time — over a persistent connection such as Server-Sent Events (SSE), WebSockets, or a chunked HTTP response.

For a local model, the generation loop already produces tokens one at a time. In a non-streaming setup, those tokens are buffered into a single string and returned at the end. In a streaming setup, each token is forwarded to the client immediately. The difference is purely in how the output is delivered, not how it is generated.

Why Streaming Matters for Local Models

Streaming is especially valuable when running models locally, for several reasons:

Perceived latency: Local models on consumer hardware can take several seconds — or tens of seconds — to produce a long response. Showing the first token in under a second dramatically improves the user experience.
Early cancellation: If the user can see output as it forms, they can stop generation as soon as they have what they need, saving compute and power.
Memory efficiency: You don't need to hold the entire response in memory on the server side before sending it.
Transparent UX: Watching text appear token-by-token gives users confidence that the model is actually working, rather than appearing frozen during a long generation.
Composability: Streaming integrates naturally with pipelines that post-process tokens, such as syntax highlighting, markdown rendering, or tool-call parsing.

How Streaming Works Under the Hood

Most autoregressive language models generate text in a loop. At each step, the model takes the current sequence, predicts a probability distribution over the next token, samples a token, appends it to the sequence, and repeats until a stop condition is met. A streaming implementation simply exposes that loop to the caller.

The typical flow is:

The client opens a connection and sends a prompt plus parameters.
The server initializes the model's KV cache and begins the generation loop.
For each new token, the server encodes it (often as JSON or plain text) and flushes it to the client.
The client decodes each chunk and appends it to the visible output.
When the model emits an end-of-sequence token or hits the max length, the server closes the stream.

Implementing Streaming with Ollama

Ollama is one of the easiest ways to run local models, and it supports streaming out of the box. When you set "stream": true in the request body, the API returns newline-delimited JSON objects, one per token.

import requests

url = "http://localhost:11434/api/generate"
payload = {
    "model": "llama3.2",
    "prompt": "Explain how transformers work in three sentences.",
    "stream": True
}

with requests.post(url, json=payload, stream=True) as resp:
    for line in resp.iter_lines():
        if not line:
            continue
        chunk = line.decode("utf-8")
        # Each line is a JSON object with a "response" field
        import json
        data = json.loads(chunk)
        print(data["response"], end="", flush=True)
print()

Each JSON object contains a "response" field with the new text, plus metadata like "done" and timing statistics. The final object has "done": true and includes the total generation duration and token count.

Implementing Streaming with llama.cpp's Python Bindings

For lower-level control, llama-cpp-python exposes a callback that fires on every token. This is ideal when you want to build your own server or integrate streaming into an existing application.

from llama_cpp import Llama

llm = Llama(
    model_path="./models/llama-3.2-3b-instruct-q4_k_m.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,  # offload all layers to GPU if available
)

def stream_callback(text: str):
    # Called once per generated token
    print(text, end="", flush=True)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a haiku about local LLMs."}],
    stream=True,
    stream_callback=stream_callback,
    max_tokens=128,
)
print()

The stream_callback receives each decoded token as a string. You can route that text anywhere — to a websocket, an SSE endpoint, a log file, or a UI component. Because the callback runs on the generation thread, keep it lightweight to avoid slowing down inference.

Implementing Streaming with Hugging Face Transformers

If you load models directly through Transformers, the TextIteratorStreamer class lets you consume tokens from a background thread. This pairs well with FastAPI or any async web framework.

import torch
from threading import Thread
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

model_id = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

def generate_stream(prompt: str):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    streamer = TextIteratorStreamer(
        tokenizer, skip_prompt=True, skip_special_tokens=True
    )
    generation_kwargs = dict(
        inputs=inputs.input_ids,
        streamer=streamer,
        max_new_tokens=256,
        temperature=0.7,
        do_sample=True,
    )
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()

    for text in streamer:
        yield text

    thread.join()

# Example usage
for token in generate_stream("List three benefits of edge AI."):
    print(token, end="", flush=True)
print()

The streamer is a Python iterator, so it composes naturally with generators. Wrapping it in a FastAPI endpoint with StreamingResponse turns this into a full HTTP streaming API.

Exposing Streaming Over HTTP with FastAPI

Server-Sent Events are the most common transport for streaming text to browsers because they work over plain HTTP and are supported natively by the EventSource API. Here is a minimal FastAPI endpoint that wraps any of the generators above.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    prompt: str

@app.post("/chat")
def chat(req: ChatRequest):
    def event_stream():
        for token in generate_stream(req.prompt):
            # SSE format: "data: <payload>\n\n"
            payload = token.replace("\n", "\\n")
            yield f"data: {payload}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        event_stream(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

The X-Accel-Buffering: no header is important if you sit behind nginx, which buffers responses by default and would otherwise defeat the purpose of streaming. On the client side, a small JavaScript snippet consumes the stream:

const source = new EventSource("/chat?prompt=hello");
source.onmessage = (event) => {
  if (event.data === "[DONE]") {
    source.close();
    return;
  }
  document.getElementById("output").textContent += event.data;
};

Note that EventSource only supports GET requests. For POST requests with a JSON body, use the fetch API and read the response body as a stream, parsing SSE frames manually.

Handling Errors and Cancellation

Streaming introduces failure modes that batch APIs do not have. A connection can drop mid-stream, the client may cancel, or the model can throw an error after several tokens have already been sent. Your implementation should handle each case gracefully.

Client disconnects: Detect a closed connection in your server loop and stop generation. With FastAPI, check request.is_disconnected() periodically. With Ollama, closing the HTTP response stream will cause the server to halt generation.
Mid-stream errors: Emit an error event in the SSE format (event: error\ndata: ...) so the client can distinguish errors from normal tokens.
Partial tokens: Some tokenizers produce bytes that are not valid UTF-8 until combined with the next token. Use skip_special_tokens=True and decode with errors="replace" or accumulate bytes until they form a complete character.
Backpressure: If the client is slow to read, buffers can grow. Use async generators and let the framework handle flow control rather than buffering everything in memory.

Best Practices

To get the most out of streaming with local models, keep these principles in mind:

Flush often. Make sure no proxy, framework, or runtime is buffering your output. Disable gzip for streaming endpoints and set the right headers.
Keep callbacks cheap. Any work done inside a token callback directly slows generation. Offload heavy processing to a separate thread or queue.
Send metadata separately. Don't pollute the text stream with timing or token counts. Use SSE event types or a final summary message.
Use a consistent chunk format. Whether you choose raw text, JSON, or SSE, stick to one schema so clients can parse reliably. A common pattern is {"token": "...", "done": false} followed by a final {"token": "", "done": true, "usage": {...}}.
Cap max tokens. Streaming does not make generation free. Set sensible max_tokens limits so a runaway model does not consume GPU time indefinitely.
Warm the model. The first request after load is slower due to lazy initialization and KV cache allocation. Send a dummy prompt at startup so the first real user request streams immediately.
Test with slow clients. Simulate a slow reader to verify backpressure handling. A model that generates faster than the client can consume should not OOM your server.
Log token timing. Track time-to-first-token and tokens-per-second separately. Time-to-first-token is the metric that most affects perceived performance in a streaming UI.

Performance Considerations

Streaming has a subtle cost: it can reduce overall throughput because the model cannot batch as aggressively when each request is being consumed token-by-token. For single-user local setups this rarely matters, but if you serve multiple concurrent users from one GPU, consider a serving engine like vLLM that supports continuous batching while still exposing a streaming API. vLLM's OpenAI-compatible server streams tokens via SSE and keeps requests batched under the hood, giving you both low latency and high throughput.

Another consideration is token decoding overhead. Decoding one token at a time is less efficient than decoding a full sequence, but the difference is negligible compared to the model forward pass. Do not prematurely optimize the decode path; focus on GPU utilization and memory bandwidth first.

Conclusion

Streaming responses transform the experience of using local models from a frustrating wait into an interactive conversation. Because local inference already produces tokens incrementally, exposing that loop to the client is mostly a matter of choosing the right transport and handling the edge cases. Whether you use Ollama for simplicity, llama.cpp for control, Transformers for flexibility, or vLLM for throughput, the patterns are the same: generate tokens in a loop, forward each one immediately, handle disconnects and errors gracefully, and keep the client informed. With the techniques in this tutorial, you can build local model applications that feel as responsive as their cloud-hosted counterparts while keeping every byte of data on your own hardware.