How to Run LLMs in the Browser with WebLLM

Running large language models (LLMs) traditionally required powerful cloud GPUs and a constant network connection. WebLLM changes that equation by bringing model inference directly into the browser using WebGPU. This tutorial walks you through what WebLLM is, why it matters, and how to integrate it into your own web applications with practical, working code examples.

What is WebLLM?

WebLLM is an open-source, high-performance in-browser LLM inference engine developed by MLC AI. It leverages the WebGPU API to execute model computations on the user's local GPU, entirely client-side. Models are downloaded once and cached, after which inference happens without any server round-trips.

Under the hood, WebLLM uses the Apache TVM compiler stack and the MLC (Machine Learning Compilation) framework to compile models like Llama 3, Phi-3, Gemma, Mistral, and Qwen into WebGPU-friendly formats. The result is a JavaScript API that feels similar to the OpenAI Chat Completion API, but runs locally.

Why It Matters

Privacy: No user prompts ever leave the device, making it ideal for sensitive workloads in healthcare, legal, or enterprise contexts.
Zero server cost: Inference runs on the client's GPU, so you don't pay for cloud compute or API tokens.
Offline capability: Once the model is cached, the application works without an internet connection.
Low latency: Without network round-trips, streaming responses can feel faster for interactive use cases.
Open ecosystem: You can swap models freely and avoid vendor lock-in.

Prerequisites

Before you begin, make sure you have the following:

A modern browser with WebGPU enabled (Chrome 113+, Edge 113+, or Safari 18+). Firefox support is still experimental.
Node.js 18+ and npm for the build tooling.
A machine with a discrete GPU or a recent integrated GPU. At least 8GB of VRAM is recommended for 7B-class models.
Basic familiarity with JavaScript and async/await.

Installing WebLLM

WebLLM ships as an npm package. Create a new project and install it:

mkdir webllm-demo && cd webllm-demo
npm init -y
npm install @mlc-ai/web-llm
npm install -D vite

We use Vite here because WebLLM relies on Web Workers and ES modules, which Vite handles cleanly out of the box.

Checking WebGPU Support

Not every browser supports WebGPU yet, so always check before initializing the engine. Create a file named index.html:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>WebLLM Demo</title>
</head>
<body>
  <h1>WebLLM Demo</h1>
  <div id="status">Checking WebGPU support...</div>
  <button id="loadBtn" disabled>Load Model</button>
  <button id="chatBtn" disabled>Send</button>
  <input id="prompt" type="text" placeholder="Ask something..." />
  <pre id="output"></pre>
  <script type="module" src="https://agentechip.com/main.js"></script>
</body>
</html>

Then create main.js with a feature check:

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const statusEl = document.getElementById("status");
const loadBtn = document.getElementById("loadBtn");
const chatBtn = document.getElementById("chatBtn");
const promptInput = document.getElementById("prompt");
const outputEl = document.getElementById("output");

if (!navigator.gpu) {
  statusEl.textContent = "WebGPU is not available in this browser.";
} else {
  statusEl.textContent = "WebGPU is available. Ready to load a model.";
  loadBtn.disabled = false;
}

Loading a Model

WebLLM exposes a CreateMLCEngine factory that downloads the model weights, compiles them for WebGPU, and prepares the engine for chat. The download can be several gigabytes, so you should report progress to the user.

const MODEL_ID = "Llama-3.2-1B-Instruct-q4f32_1-MLC";

let engine = null;

loadBtn.addEventListener("click", async () => {
  loadBtn.disabled = true;
  statusEl.textContent = "Loading model... this may take a while.";

  const initProgressCallback = (report) => {
    statusEl.textContent =
      `Loading: ${report.text} ` +
      `(${(report.progress * 100).toFixed(1)}%)`;
  };

  try {
    engine = await CreateMLCEngine(
      MODEL_ID,
      { initProgressCallback }
    );
    statusEl.textContent = "Model loaded and ready.";
    chatBtn.disabled = false;
  } catch (err) {
    statusEl.textContent = "Failed to load model: " + err.message;
    loadBtn.disabled = false;
  }
});

The MODEL_ID string maps to a precompiled model hosted on Hugging Face. Smaller quantized variants like Llama-3.2-1B-Instruct-q4f32_1-MLC are great for demos because they download quickly and fit in modest VRAM. For higher quality, try Llama-3.1-8B-Instruct-q4f32_1-MLC or Phi-3.5-mini-instruct-q4f16_1-MLC.

Running Chat Completions

Once the engine is ready, you can call engine.chat.completions.create() with a message array. The API mirrors the OpenAI SDK, so existing prompt patterns translate directly.

chatBtn.addEventListener("click", async () => {
  if (!engine) return;
  const userPrompt = promptInput.value.trim();
  if (!userPrompt) return;

  outputEl.textContent = "";
  chatBtn.disabled = true;

  const messages = [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: userPrompt },
  ];

  const chunks = await engine.chat.completions.create({
    messages,
    stream: true,
    temperature: 0.7,
    max_tokens: 512,
  });

  let fullText = "";
  for await (const chunk of chunks) {
    const delta = chunk.choices[0]?.delta?.content || "";
    fullText += delta;
    outputEl.textContent = fullText;
  }

  chatBtn.disabled = false;
});

Streaming is important for UX: it lets the user see tokens as they are generated, which feels much more responsive than waiting for the entire response to complete. The async iterator pattern shown above is the recommended way to consume the stream.

Using a Web Worker for Better Performance

Running inference on the main thread can block the UI and cause janky animations. WebLLM provides CreateWebWorkerMLCEngine to move the heavy lifting into a dedicated worker.

Create worker.js:

import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => handler.onmessage(msg);

Then update main.js to use the worker variant:

import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm";

const worker = new Worker(
  new URL("./worker.js", import.meta.url),
  { type: "module" }
);

engine = await CreateWebWorkerMLCEngine(
  worker,
  MODEL_ID,
  { initProgressCallback }
);

The rest of your chat code stays identical because the worker engine exposes the same interface. This is the recommended setup for production applications.

Managing Conversation State

WebLLM keeps a running KV cache on the engine, so multi-turn conversations work naturally. You can append messages and call create() again without re-sending the full history each time, but for clarity many apps prefer to manage the message array explicitly:

const conversation = [
  { role: "system", content: "You are a concise coding assistant." },
];

function addUserMessage(text) {
  conversation.push({ role: "user", content: text });
}

async function respond() {
  const chunks = await engine.chat.completions.create({
    messages: conversation,
    stream: true,
  });

  let reply = "";
  for await (const chunk of chunks) {
    reply += chunk.choices[0]?.delta?.content || "";
  }
  conversation.push({ role: "assistant", content: reply });
  return reply;
}

// Reset when starting a new topic
async function resetConversation() {
  await engine.resetChat();
  conversation.length = 1; // keep only the system message
}

Calling engine.resetChat() clears the internal KV cache, which is important when switching topics to avoid stale context bleeding into new answers.

Choosing the Right Model

Model selection is a trade-off between quality, download size, and VRAM usage. Here are practical guidelines:

1B–3B models (Llama-3.2-1B, Qwen2.5-1.5B): Fast, small downloads (~1GB), run on most laptops. Good for simple Q&A and demos.
7B–8B models (Llama-3.1-8B, Mistral-7B): Strong general quality, ~4GB downloads, need 6GB+ VRAM. The sweet spot for most apps.
Specialized models (Phi-3.5-mini, Gemma-2): Often punch above their size for specific tasks like reasoning or instruction following.
Quantization: Prefer q4f16_1 or q4f32_1 variants. They cut memory usage dramatically with minimal quality loss.

Best Practices

Always check WebGPU availability first and provide a clear fallback message or a server-based alternative.
Show download progress. First-time loads can take minutes on slow connections. A progress bar prevents users from thinking the app is frozen.
Use a Web Worker. Keep the main thread free for rendering and interaction.
Cache aggressively. WebLLM uses the browser Cache API automatically, so repeat visits skip the download. Inform users of this so they don't clear storage unnecessarily.
Set sensible defaults. Limit max_tokens for chat to avoid runaway generations, and pick a moderate temperature unless you specifically need creativity.
Handle interruptions. Call engine.interruptGenerate() when the user clicks a stop button so they don't have to wait for a long response to finish.
Test on real hardware. Performance varies wildly between integrated and discrete GPUs. Profile on low-end devices before shipping.
Be honest about limits. Browser LLMs are smaller than frontier cloud models. Don't promise GPT-4-level reasoning from a 1B model.

Stopping Generation Early

For a stop button, wire up an interrupt handler:

const stopBtn = document.getElementById("stopBtn");
let generating = false;

stopBtn.addEventListener("click", () => {
  if (engine && generating) {
    engine.interruptGenerate();
  }
});

async function generate(messages) {
  generating = true;
  try {
    const chunks = await engine.chat.completions.create({
      messages,
      stream: true,
    });
    let text = "";
    for await (const chunk of chunks) {
      text += chunk.choices[0]?.delta?.content || "";
      outputEl.textContent = text;
    }
    return text;
  } finally {
    generating = false;
  }
}

Conclusion

WebLLM makes it genuinely practical to run capable language models entirely in the browser, opening the door to private, offline, and zero-cost AI experiences. By combining WebGPU acceleration, a familiar OpenAI-style API, and Web Worker support, it fits cleanly into modern web workflows. Start with a small quantized model to validate your UX, then scale up to 7B-class models once you've confirmed your target hardware can handle them. With thoughtful progress reporting, graceful fallbacks, and worker-based inference, you can ship browser-native LLM features that feel fast, private, and reliable.