Building a Desktop App with Tauri and Local LLMs

Building a Desktop App with Tauri and Local LLMs

Running large language models locally inside a desktop application used to be a niche experiment. With the rise of efficient inference engines like llama.cpp, ollama, and ONNX Runtime, plus the maturation of Tauri as a lightweight alternative to Electron, it's now practical to ship a fully offline AI assistant as a single installer under 20 MB (model weights aside). This tutorial walks through the entire process: scaffolding a Tauri app, embedding a local LLM, wiring up a chat UI, and packaging the result.

What You're Building

The end product is a cross-platform desktop app — Windows, macOS, and Linux — that loads a quantized GGUF model from disk, streams responses token-by-token into a chat interface, and never sends a single byte to the cloud. The frontend is plain HTML/CSS/JS (or any framework you prefer), and the Rust backend handles model loading, tokenization, and inference.

Why This Stack Matters

Electron-based AI apps bundle a full Chromium instance, often pushing installers past 150 MB before you've added a model. Tauri uses the OS-native webview, cutting binary size dramatically and reducing memory overhead. Pairing that with a local LLM gives you four concrete advantages:

• Privacy by design. Prompts and completions never leave the machine. This matters for legal, medical, and enterprise use cases where data residency is non-negotiable.
• Zero recurring cost. No API keys, no per-token billing, no rate limits. You pay once in compute.
• Offline operation. The app works on airplanes, in secure facilities, and behind corporate firewalls.
• Deterministic deployment. You control the exact model version, quantization, and inference parameters. No silent upstream changes.

The tradeoff is hardware: you need enough RAM to hold the model. A 4-bit quantized 7B parameter model fits comfortably in 6–8 GB of RAM, while a 3B model runs on machines with 4 GB. We'll design the app to detect available memory and recommend an appropriate model.

Prerequisites and Project Setup

Install the Tauri prerequisites first. You need Rust (stable), Node.js 18+, and the platform build tools:

• Windows: Microsoft Visual Studio C++ Build Tools and WebView2 (preinstalled on Windows 11).
• macOS: Xcode Command Line Tools (xcode-select --install).
• Linux: libwebkit2gtk-4.1-dev, build-essential, libssl-dev, and related GTK packages.

Scaffold a new Tauri project with the vanilla template:

npm create tauri-app@latest local-llm-desktop
cd local-llm-desktop
npm install
npm run tauri dev

Confirm the default window opens. Now add the crates that power local inference. We'll use candle-core and candle-transformers from Hugging Face, which are pure-Rust and integrate cleanly with Tauri's build system. Alternatively, you can shell out to ollama as a subprocess — simpler but adds an external dependency.

Edit src-tauri/Cargo.toml:

[package]
name = "local-llm-desktop"
version = "0.1.0"
edition = "2021"

[dependencies]
tauri = { version = "1.6", features = ["shell-open"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
candle-core = "0.4"
candle-transformers = "0.4"
candle-nn = "0.4"
tokenizers = "0.15"
hf-hub = "0.3"
tokio = { version = "1", features = ["full"] }
anyhow = "1"

[build-dependencies]
tauri-build = { version = "1.6" }

Loading a Model in Rust

Create src-tauri/src/llm.rs to encapsulate model loading and generation. We'll target a small instruct-tuned model like Qwen2.5-1.5B-Instruct quantized to GGUF or loaded directly via candle's safetensors path. For simplicity, this example uses the safetensors path through hf-hub.

use anyhow::{Context, Result};
use candle_core::{Device, Tensor};
use candle_transformers::models::qwen2::Config;
use candle_transformers::models::qwen2::Model as QwenModel;
use candle_nn::VarBuilder;
use hf_hub::{api::sync::Api, Repo, RepoType};
use tokenizers::Tokenizer;
use std::sync::Mutex;

pub struct LlmEngine {
    pub model: Mutex<QwenModel>,
    pub tokenizer: Tokenizer,
    pub device: Device,
}

impl LlmEngine {
    pub fn load(model_id: &str, revision: &str) -> Result<Self> {
        let api = Api::new()?;
        let repo = api.repo(Repo::with_revision(
            model_id.to_string(),
            RepoType::Model,
            revision.to_string(),
        ));

        let tokenizer_file = repo.get("tokenizer.json")?;
        let tokenizer = Tokenizer::from_file(tokenizer_file)
            .map_err(|e| anyhow::anyhow!("tokenizer error: {e:?}"))?;

        let weights = repo.get("model.safetensors")?;
        let device = Device::Cpu;
        let vb = VarBuilder::from_mmaped_safetensors(&[weights], candle_core::DType::F32, &device)?;
        let config = Config::default();
        let model = QwenModel::new(&config, vb)?;

        Ok(Self {
            model: Mutex::new(model),
            tokenizer,
            device,
        })
    }

    pub fn generate(&self, prompt: &str, max_tokens: usize) -> Result<String> {
        let tokens = self
            .tokenizer
            .encode(prompt, true)
            .map_err(|e| anyhow::anyhow!("encode error: {e:?}"))?
            .get_ids()
            .to_vec();

        let mut input = Tensor::new(tokens.as_slice(), &self.device)?
            .unsqueeze(0)?;

        let mut generated = String::new();
        let mut model = self.model.lock().unwrap();

        for _ in 0..max_tokens {
            let logits = model.forward(&input)?;
            let next_token = logits.argmax(1)?.get(0).to_vec::<u32>()?[0];
            if next_token as usize == 0 {
                break;
            }
            let piece = self
                .tokenizer
                .decode(&[next_token], true)
                .map_err(|e| anyhow::anyhow!("decode error: {e:?}"))?;
            generated.push_str(&piece);
            input = Tensor::new(&[next_token], &self.device)?.unsqueeze(0)?;
        }

        Ok(generated)
    }
}

This is a minimal greedy decoder. For production you'd add temperature sampling, KV-cache reuse, and stop-token handling, but it demonstrates the core loop: tokenize, forward, argmax, decode, append.

Exposing Commands to the Frontend

Tauri's command system lets the webview call Rust functions directly. Wire up two commands: one to initialize the engine at startup, one to generate text. Open src-tauri/src/main.rs:

#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]

mod llm;

use llm::LlmEngine;
use std::sync::Arc;
use tauri::State;

struct AppState {
    engine: Arc<LlmEngine>,
}

#[tauri::command]
fn generate(prompt: String, state: State<AppState>) -> Result<String, String> {
    state
        .engine
        .generate(&prompt, 256)
        .map_err(|e| e.to_string())
}

fn main() {
    let model_id = "Qwen/Qwen2.5-1.5B-Instruct";
    let revision = "main".to_string();

    let engine = LlmEngine::load(model_id, &revision)
        .expect("failed to load model");

    tauri::Builder::default()
        .manage(AppState {
            engine: Arc::new(engine),
        })
        .invoke_handler(tauri::generate_handler![generate])
        .run(tauri::generate_context!())
        .expect("error while running tauri application");
}

On first launch, the app downloads the model from the Hugging Face Hub and caches it under ~/.cache/huggingface. Subsequent launches load from disk in a few seconds. If you want a fully offline installer, bundle the weights in src-tauri/resources/ and reference them with tauri::path::resolve_resource.

Building the Chat UI

Replace index.html with a minimal chat interface. We'll keep dependencies to zero — no React, no Tailwind — to honor Tauri's lightweight philosophy.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <title>Local LLM Desktop</title>
  <style>
    body { font-family: system-ui; margin: 0; display: flex; flex-direction: column; height: 100vh; }
    #log { flex: 1; overflow-y: auto; padding: 16px; }
    .msg { margin-bottom: 12px; padding: 10px; border-radius: 8px; max-width: 80%; }
    .user { background: #2563eb; color: white; margin-left: auto; }
    .bot { background: #e5e7eb; color: black; }
    form { display: flex; padding: 12px; gap: 8px; border-top: 1px solid #ddd; }
    input { flex: 1; padding: 10px; font-size: 14px; border: 1px solid #ccc; border-radius: 6px; }
    button { padding: 10px 20px; background: #2563eb; color: white; border: none; border-radius: 6px; cursor: pointer; }
  </style>
</head>
<body>
  <div id="log"></div>
  <form id="chat-form">
    <input id="prompt" placeholder="Ask anything..." autocomplete="off" />
    <button type="submit">Send</button>
  </form>
  <script src="main.js"></script>
</body>
</html>

Then main.js handles the Tauri invoke and appends messages to the log:

const { invoke } = window.__TAURI__.tauri;

const log = document.getElementById("log");
const form = document.getElementById("chat-form");
const promptInput = document.getElementById("prompt");

function appendMessage(text, role) {
  const div = document.createElement("div");
  div.className = `msg ${role}`;
  div.textContent = text;
  log.appendChild(div);
  log.scrollTop = log.scrollHeight;
}

form.addEventListener("submit", async (e) => {
  e.preventDefault();
  const prompt = promptInput.value.trim();
  if (!prompt) return;

  appendMessage(prompt, "user");
  promptInput.value = "";
  promptInput.disabled = true;

  try {
    const response = await invoke("generate", { prompt });
    appendMessage(response, "bot");
  } catch (err) {
    appendMessage(`Error: ${err}`, "bot");
  } finally {
    promptInput.disabled = false;
    promptInput.focus();
  }
});

Run npm run tauri dev again. The first invocation takes longer as the model warms up; subsequent prompts respond in a few hundred milliseconds on a modern CPU.

Streaming Responses

Blocking on a full completion feels sluggish for longer answers. Tauri's event system lets you stream tokens from Rust to the frontend. Update the generate command to emit events instead of returning a single string:

use tauri::{AppHandle, Emitter, State};

#[tauri::command]
async fn generate_stream(
    prompt: String,
    app: AppHandle,
    state: State<'_, AppState>,
) -> Result<(), String> {
    let engine = state.engine.clone();
    tokio::task::spawn_blocking(move || -> Result<(), String> {
        let tokens = engine
            .tokenizer
            .encode(&prompt, true)
            .map_err(|e| e.to_string())?
            .get_ids()
            .to_vec();

        let mut input = Tensor::new(tokens.as_slice(), &engine.device)
            .map_err(|e| e.to_string())?
            .unsqueeze(0)
            .map_err(|e| e.to_string())?;

        let mut model = engine.model.lock().unwrap();
        for _ in 0..256 {
            let logits = model.forward(&input).map_err(|e| e.to_string())?;
            let next = logits.argmax(1).map_err(|e| e.to_string())?
                .get(0).to_vec::<u32>().map_err(|e| e.to_string())?[0];
            if next == 0 { break; }
            let piece = engine.tokenizer
                .decode(&[next], true)
                .map_err(|e| e.to_string())?;
            let _ = app.emit("token", &piece);
            input = Tensor::new(&[next], &engine.device)
                .map_err(|e| e.to_string())?
                .unsqueeze(0).map_err(|e| e.to_string())?;
        }
        let _ = app.emit("done", ());
        Ok(())
    })
    .await
    .map_err(|e| e.to_string())?
}

On the frontend, listen for those events and append incrementally:

const { listen } = window.__TAURI__.event;

let currentBotMsg = null;

async function setupStream() {
  await listen("token", (event) => {
    if (!currentBotMsg) {
      currentBotMsg = document.createElement("div");
      currentBotMsg.className = "msg bot";
      log.appendChild(currentBotMsg);
    }
    currentBotMsg.textContent += event.payload;
    log.scrollTop = log.scrollHeight;
  });

  await listen("done", () => {
    currentBotMsg = null;
    promptInput.disabled = false;
    promptInput.focus();
  });
}

setupStream();

form.addEventListener("submit", async (e) => {
  e.preventDefault();
  const prompt = promptInput.value.trim();
  if (!prompt) return;
  appendMessage(prompt, "user");
  promptInput.value = "";
  promptInput.disabled = true;
  await invoke("generate_stream", { prompt });
});

Best Practices

• Choose the smallest model that works. A 1.5B or 3B instruct model handles most assistant tasks and keeps memory pressure low. Scale up only when quality demands it.
• Use quantization. GGUF Q4_K_M or Q5_K_M cuts memory by 4–8x with negligible quality loss. Candle supports GGUF via the candle-core GGML integration.
• Run inference off the main thread. Always wrap generation in spawn_blocking or a dedicated thread. Blocking the Tauri event loop freezes the UI.
• Implement stop tokens and length limits. Without them, the model can ramble indefinitely. Check the tokenizer's special token IDs and break the loop when you hit EOS.
• Cache the KV cache. For multi-turn chat, reuse the key-value cache from previous turns instead of re-encoding the full history each time. This is the single biggest performance win.
• Surface hardware limits gracefully. Detect available RAM via sysinfo and warn users before loading a model that won't fit. A crash on OOM is a poor first impression.
• Bundle or stream, don't both. Decide upfront whether the model ships with the installer (large download, instant offline use) or downloads on first run (small installer, requires internet once). Mixing the two confuses users.
• Sanitize prompts for system messages. Even local models can be jailbroken. If you prepend a system prompt, treat user input as untrusted data, not instructions.

Packaging and Distribution

Build a production installer with:

npm run tauri build

This produces platform-specific bundles in src-tauri/target/release/bundle/ — MSI and EXE on Windows, DMG and APP on macOS, AppImage and DEB on Linux. Sign the binaries with your platform's tooling before distribution; unsigned installers trigger scary warnings that erode trust.

If you bundle model weights, list them in tauri.conf.json under tauri.bundle.resources so they're included in the installer:

{
  "tauri": {
    "bundle": {
      "resources": ["resources/*.gguf"]
    }
  }
}

Conclusion

Tauri plus a local LLM gives you a privacy-preserving, offline-capable desktop assistant in a package that respects your users' disk space and memory. The core loop — load weights in Rust, expose a command, stream tokens to a webview — is straightforward, and the hard work lives in choosing the right model, tuning inference parameters, and building a UI that feels responsive even when generation takes a few seconds. Start with a small quantized model, get the streaming pipeline solid, then iterate on quality by swapping in larger models as your hardware budget allows. The result is an app your users can run anywhere, trust completely, and own outright.