Streaming Structured JSON from LLMs to the Frontend

Streaming Structured JSON from LLMs to the Frontend

Large language models are powerful, but they're also slow. When you ask an LLM to generate a long structured response — say, a list of products, a multi-section report, or a complex JSON object — the user can wait several seconds (or longer) staring at a spinner before anything appears. Streaming solves this latency problem by sending tokens to the client as they're generated. But streaming structured data like JSON introduces a unique challenge: how do you parse and render JSON that isn't finished yet?

This tutorial walks through the full stack of streaming structured JSON from an LLM to a frontend application. We'll cover the core concepts, build a working backend with Server-Sent Events, implement a partial-JSON parser on the frontend, and discuss best practices for production use.

What Is Structured JSON Streaming?

Structured JSON streaming is the practice of sending a JSON payload from an LLM to a client incrementally, token by token, while still allowing the client to parse and use the partial result before the full document is complete. Unlike plain text streaming — where you can simply append each chunk to a string and render it — JSON streaming requires you to handle incomplete syntax gracefully.

Consider this scenario: you ask an LLM to return a list of articles with titles, summaries, and tags. The model begins generating:

{
  "articles": [
    { "title": "Understanding React Server Components", "summary": "A deep dive

At this point, the JSON is syntactically invalid. A standard JSON.parse() call would throw. But a human reading this can already see useful information — there's an article about React Server Components. The goal of structured streaming is to let your frontend surface that information immediately, updating the UI as more tokens arrive, rather than waiting for the closing braces.

Why It Matters

There are three major reasons structured JSON streaming has become an important pattern in modern AI application development:

• Perceived performance: Users see results appear progressively. A response that takes 8 seconds to fully generate can start showing useful content in under 1 second, dramatically improving the user experience.
• Interactivity during generation: When partial results are parseable, the frontend can begin downstream work — fetching related data, pre-rendering components, or letting the user interact with early results — while the LLM is still generating.
• Graceful degradation: If a connection drops mid-stream, you still have the partial structured data that arrived. With plain text streaming, you'd have an incomplete string with no semantic structure.

For applications like AI-powered dashboards, code generators, or data extraction tools, this pattern can be the difference between a product that feels instant and one that feels sluggish.

The Core Challenge: Parsing Incomplete JSON

The fundamental problem is that JSON.parse() is all-or-nothing. It either succeeds with a complete document or throws a SyntaxError. There's no built-in mechanism in JavaScript to parse a partial JSON string and return what it can.

Several approaches exist to solve this:

• Manual bracket tracking: Walk the string character by character, tracking open brackets and braces, and attempt a parse whenever you detect a complete top-level element. This works for arrays of objects but is fragile for nested structures.
• Repair-based parsing: Use a library that "repairs" incomplete JSON by closing open brackets and braces, filling in missing values, and then parsing. Libraries like partial-json and jsonrepair do exactly this.
• Streaming JSON parsers: Use a SAX-style or streaming parser like stream-json or clarinet that emits events as it encounters tokens, rather than waiting for the full document.

For most frontend use cases, the repair-based approach offers the best balance of simplicity and robustness. Let's build a complete example using that strategy.

Building the Backend: Streaming with Server-Sent Events

Our backend will call an LLM API (we'll use OpenAI's API as an example), request a structured JSON response, and stream the tokens to the frontend using Server-Sent Events (SSE). SSE is ideal here because it's a one-way streaming protocol that works over standard HTTP and is supported by the browser's EventSource API.

Setting Up the Express Server

First, install the necessary dependencies:

npm install express openai cors

Then create your server file. The key elements are setting the correct SSE headers, calling the LLM with stream: true, and forwarding each token chunk to the client:

import express from "express";
import cors from "cors";
import OpenAI from "openai";

const app = express();
app.use(cors());
app.use(express.json());

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

app.post("/api/generate", async (req, res) => {
  // Set SSE headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders();

  const { topic } = req.body;

  try {
    const stream = await openai.chat.completions.create({
      model: "gpt-4o",
      stream: true,
      response_format: { type: "json_object" },
      messages: [
        {
          role: "system",
          content: `You are a helpful assistant. Respond with a JSON object containing an "articles" array. Each article has "title", "summary", and "tags" fields. Generate 5 articles about the given topic.`
        },
        {
          role: "user",
          content: `Topic: ${topic}`
        }
      ],
    });

    for await (const chunk of stream) {
      const token = chunk.choices[0]?.delta?.content;
      if (token) {
        // Send each token as an SSE data event
        res.write(`data: ${JSON.stringify({ token })}\n\n`);
      }
    }

    // Signal completion
    res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
    res.end();
  } catch (error) {
    console.error("Stream error:", error);
    res.write(`data: ${JSON.stringify({ error: error.message })}\n\n`);
    res.end();
  }
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

Notice the response_format: { type: "json_object" } option. This instructs the model to produce valid JSON, which makes our partial parsing much more reliable. The model still streams token by token, but the overall structure is constrained to JSON.

Why SSE Instead of WebSockets?

WebSockets are bidirectional, which is more than we need for a simple request-response streaming pattern. SSE is simpler, automatically reconnects on connection loss, and works through proxies and firewalls more reliably. For LLM streaming where the client sends one request and receives a stream of tokens back, SSE is the standard choice. OpenAI, Anthropic, and most LLM providers use SSE for their own streaming APIs for the same reasons.

Building the Frontend: Consuming and Parsing the Stream

On the frontend, we need to do three things: connect to the SSE endpoint, accumulate the incoming tokens, and parse the partial JSON on each update so we can render progressively.

Connecting to the Stream

While the browser's EventSource API is convenient, it only supports GET requests. Since our endpoint is a POST, we'll use the fetch API with a streaming response reader instead. This gives us full control over the request body and headers:

async function streamArticles(topic, onToken, onParsed, onDone, onError) {
  try {
    const response = await fetch("http://localhost:3000/api/generate", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ topic }),
    });

    if (!response.ok) {
      throw new Error(`HTTP ${response.status}`);
    }

    const reader = response.body.getReader();
    const decoder = new TextDecoder();
    let buffer = "";
    let fullText = "";

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });

      // SSE events are separated by double newlines
      const lines = buffer.split("\n\n");
      buffer = lines.pop(); // Keep incomplete chunk in buffer

      for (const line of lines) {
        if (!line.startsWith("data: ")) continue;
        const data = JSON.parse(line.slice(6));

        if (data.error) {
          onError(data.error);
          return;
        }
        if (data.done) {
          onDone();
          return;
        }
        if (data.token) {
          fullText += data.token;
          onToken(data.token);
          onParsed(fullText);
        }
      }
    }
  } catch (err) {
    onError(err.message);
  }
}

Parsing Partial JSON

Now for the key piece: parsing the accumulated fullText as it grows. We'll use the partial-json library, which attempts to parse incomplete JSON and returns whatever it can. Install it first:

npm install partial-json

Then create a utility function that safely parses the partial string:

import { parse } from "partial-json";

function parsePartialJson(text) {
  try {
    // The second argument is an allowance for partial parsing
    return parse(text, true);
  } catch (e) {
    return null;
  }
}

The parse function from partial-json handles incomplete objects, arrays, and strings. For example, given the input {"articles": [{"title": "Hello", "summary": "Wor, it will return:

{
  articles: [
    { title: "Hello", summary: "Wor" }
  ]
}

The incomplete string value "Wor is still returned (with the unclosed quote handled), and the open objects and arrays are closed automatically. This lets you render the UI with whatever data is available.

Putting It All Together in a React Component

Here's a complete React component that ties everything together. It sends a request, parses the partial JSON on every token, and renders the articles list as it grows:

import { useState, useCallback } from "react";
import { parse } from "partial-json";

function ArticleGenerator() {
  const [topic, setTopic] = useState("");
  const [articles, setArticles] = useState([]);
  const [isStreaming, setIsStreaming] = useState(false);
  const [error, setError] = useState(null);

  const handleGenerate = useCallback(async () => {
    setIsStreaming(true);
    setError(null);
    setArticles([]);

    try {
      const response = await fetch("http://localhost:3000/api/generate", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ topic }),
      });

      const reader = response.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";
      let fullText = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split("\n\n");
        buffer = lines.pop();

        for (const line of lines) {
          if (!line.startsWith("data: ")) continue;
          const data = JSON.parse(line.slice(6));

          if (data.error) {
            setError(data.error);
            break;
          }
          if (data.done) continue;
          if (data.token) {
            fullText += data.token;

            // Parse the partial JSON and update state
            try {
              const parsed = parse(fullText, true);
              if (parsed?.articles) {
                setArticles(parsed.articles);
              }
            } catch {
              // Ignore parse errors for incomplete chunks
            }
          }
        }
      }
    } catch (err) {
      setError(err.message);
    } finally {
      setIsStreaming(false);
    }
  }, [topic]);

  return (
    <div>
      <h1>Article Generator</h1>
      <input
        type="text"
        value={topic}
        onChange={(e) => setTopic(e.target.value)}
        placeholder="Enter a topic..."
      />
      <button onClick={handleGenerate} disabled={isStreaming || !topic}>
        {isStreaming ? "Generating..." : "Generate"}
      </button>

      {error && <p style={{ color: "red" }}>{error}</p>}

      <ul>
        {articles.map((article, index) => (
          <li key={index}>
            <h3>{article.title || "..."}</h3>
            <p>{article.summary || ""}</p>
            {article.tags && (
              <div>
                {article.tags.map((tag, i) => (
                  <span key={i} className="tag">{tag}</span>
                ))}
              </div>
            )}
          </li>
        ))}
      </ul>
    </div>
  );
}

export default ArticleGenerator;

As the LLM generates tokens, you'll see articles appear one at a time. Each article's title appears first, then the summary fills in word by word, and finally the tags populate. The user gets immediate feedback and can start reading the first article while the rest are still being generated.

Handling Edge Cases

Throttling State Updates

LLMs can emit tokens very rapidly — sometimes dozens per second. Calling setArticles on every single token can cause excessive re-renders. A simple throttle or debounce improves performance significantly:

let lastUpdate = 0;
const THROTTLE_MS = 50;

// Inside the token handling loop:
fullText += data.token;
const now = Date.now();
if (now - lastUpdate > THROTTLE_MS) {
  lastUpdate = now;
  try {
    const parsed = parse(fullText, true);
    if (parsed?.articles) {
      setArticles(parsed.articles);
    }
  } catch {}
}

This limits updates to at most 20 per second, which is smooth enough for the human eye while avoiding render thrashing.

Handling Disconnections

Network connections can drop. With the fetch-based approach, a dropped connection will cause the reader to throw or return done: true prematurely. You should always preserve the partial data you've received so far. The pattern above already does this — articles state retains whatever was parsed before the disconnection. You can add a retry mechanism that sends the original request again, or simply show the partial results with a "connection interrupted" notice.

Dealing with Non-JSON Prefixes

Some models, despite instructions to output only JSON, prepend text like "Here is the JSON:" or wrap the output in markdown code fences. This will break your partial parser. Two strategies help:

• Use the model's native structured output mode (like OpenAI's response_format or Anthropic's tool use) to guarantee JSON output.
• On the frontend, strip any text before the first { or [ character before attempting to parse.

function extractJsonStart(text) {
  const objIndex = text.indexOf("{");
  const arrIndex = text.indexOf("[");
  if (objIndex === -1 && arrIndex === -1) return text;
  if (objIndex === -1) return text.slice(arrIndex);
  if (arrIndex === -1) return text.slice(objIndex);
  return text.slice(Math.min(objIndex, arrIndex));
}

// Usage:
const jsonText = extractJsonStart(fullText);
const parsed = parse(jsonText, true);

Best Practices

• Use structured output modes when available. OpenAI's response_format: { type: "json_object" } or json_schema mode, and similar features in other providers, dramatically reduce parsing failures by constraining the model to valid JSON.
• Keep JSON shallow when possible. Deeply nested structures are harder to parse partially and slower to become useful. Flatten your schema where it makes sense — an array of flat objects streams better than a deeply nested tree.
• Design schemas for progressive usefulness. Put the most important fields first in your schema. LLMs tend to generate fields in the order they appear in the schema or instructions, so front-loading critical fields means the user sees valuable data sooner.
• Throttle frontend updates. As shown above, limit re-renders to a reasonable frequency. Parsing partial JSON on every token is computationally cheap, but triggering React re-renders 50 times per second is wasteful.
• Always handle the error case. Wrap parse calls in try-catch, handle SSE error events, and provide fallback UI. Partial JSON parsing is inherently heuristic — it will occasionally fail on unusual token boundaries.
• Consider streaming individual array elements. If your response is a large array, you can parse for complete elements and append them to your state one at a time, rather than re-parsing and re-rendering the entire array on each token. This is more efficient for very large responses.
• Test with different models. Tokenization differs between models, and some models produce JSON with different formatting (pretty-printed vs. compact). Test your parser against the specific models you plan to use in production.

Alternative: Using Vercel AI SDK

If you're building a React or Next.js application, the Vercel AI SDK abstracts away much of this complexity. Its streamObject function handles structured streaming end-to-end, including partial parsing:

// Server-side (Next.js route handler)
import { streamObject } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";

export async function POST(req) {
  const { topic } = await req.json();

  const result = streamObject({
    model: openai("gpt-4o"),
    schema: z.object({
      articles: z.array(
        z.object({
          title: z.string(),
          summary: z.string(),
          tags: z.array(z.string()),
        })
      ),
    }),
    prompt: `Generate 5 articles about: ${topic}`,
  });

  return result.toTextStreamResponse();
}

// Client-side (React component)
import { useObject } from "ai/react";
import { z } from "zod";

const schema = z.object({
  articles: z.array(
    z.object({
      title: z.string(),
      summary: z.string(),
      tags: z.array(z.string()),
    })
  ),
});

function ArticleGenerator() {
  const { object, submit, isLoading } = useObject({
    api: "/api/generate",
    schema,
  });

  return (
    <div>
      <button onClick={() => submit({ topic: "Machine Learning" })}>
        Generate
      </button>
      <ul>
        {object?.articles?.map((article, i) => (
          <li key={i}>
            <h3>{article.title}</h3>
            <p>{article.summary}</p>
          </li>
        ))}
      </ul>
    </div>
  );
}

The AI SDK handles the SSE transport, partial JSON parsing, and React state management for you. The object returned by useObject is always a partially-parsed version of the schema, updating in real time as tokens arrive. For production applications, using a battle-tested library like this is often preferable to rolling your own implementation.

Conclusion

Streaming structured JSON from LLMs to the frontend is a technique that transforms the user experience of AI-powered applications. By combining Server-Sent Events on the backend, partial JSON parsing on the frontend, and thoughtful schema design, you can make even complex LLM responses feel instantaneous. The key principles are straightforward: use structured output modes to guarantee JSON from the model, stream tokens over SSE, parse partial JSON progressively with a repair-based library, throttle your UI updates to avoid render thrashing, and always handle errors gracefully. Whether you implement the plumbing yourself or use a higher-level library like the Vercel AI SDK, the pattern is the same — and the payoff in perceived performance is well worth the effort. As LLMs continue to power more of the applications we build, streaming structured data will remain an essential tool in every developer's toolkit.