Building a Voice Assistant Backend with llama.cpp: Complete Guide

Building a Voice Assistant Backend with llama.cpp: Complete Guide

Voice assistants have moved from novelty to necessity, powering everything from smart home devices to enterprise customer support. While cloud-based solutions like OpenAI's Whisper and GPT-4 dominate the conversation, there's a growing demand for local, private, and cost-effective alternatives. Enter llama.cpp — a C++ inference engine that lets you run large language models on commodity hardware without GPU clusters or API bills.

This tutorial walks you through building a complete voice assistant backend using llama.cpp as the reasoning engine. You'll learn how to wire together speech-to-text, LLM inference, and text-to-speech into a single service that runs entirely on your own infrastructure.

What is llama.cpp and Why It Matters

llama.cpp is an open-source C/C++ library developed by Georgi Gerganov that enables efficient inference of LLaMA-style models on CPUs and GPUs. It supports quantized models (GGUF format), which dramatically reduce memory requirements while maintaining reasonable quality. A model that needs 16GB in full precision can run in 4GB with 4-bit quantization.

Why Choose llama.cpp for Voice Assistants?

• Privacy: No data leaves your server. Audio and transcripts never touch third-party APIs.
• Cost: No per-token billing. Run inference as much as your hardware allows.
• Latency: Local inference eliminates network round-trips, critical for conversational responsiveness.
• Flexibility: Swap models freely — Llama 3, Mistral, Phi, Qwen, and many others work out of the box.
• Portability: Runs on Raspberry Pi, laptops, workstations, and dedicated servers alike.

Architecture Overview

A voice assistant backend has three core components arranged in a pipeline:

[Audio Input] → [STT Engine] → [LLM (llama.cpp)] → [TTS Engine] → [Audio Output]

For this tutorial, we'll use:

• STT: whisper.cpp — the same C++ philosophy applied to OpenAI's Whisper model
• LLM: llama.cpp with a quantized Llama 3 8B model
• TTS: piper — a fast neural TTS engine that runs locally
• Orchestration: Python with FastAPI for the HTTP layer

The Python layer communicates with the C++ engines through their respective Python bindings or subprocess calls. This gives you the performance of native code with the developer ergonomics of Python.

Prerequisites and Setup

System Requirements

• Linux or macOS (Windows works with WSL2)
• 8GB RAM minimum (16GB recommended for 8B models)
• Python 3.10+
• Build essentials: gcc, make, cmake
• Optional: NVIDIA GPU with CUDA for faster inference

Installing llama.cpp

Build llama.cpp from source to get the latest features and optimizations:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make

# For CUDA support:
make GGML_CUDA=1

# Verify the build
./llama-cli --version

Installing whisper.cpp

git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make

# Download the base model
bash ./models/download-ggml-model.sh base.en

Installing Piper TTS

# Download pre-built binary
wget https://github.com/rhasspy/piper/releases/latest/download/piper_linux_x86_64.tar.gz
tar -xzf piper_linux_x86_64.tar.gz
sudo mv piper /usr/local/bin/

# Download a voice model
wget https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx

Python Dependencies

pip install fastapi uvicorn pydantic python-multipart httpx
pip install llama-cpp-python

If you have a CUDA GPU, install llama-cpp-python with GPU support:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Downloading the LLM Model

Download a quantized Llama 3 8B model in GGUF format:

wget https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf

The Q4_K_M quantization offers an excellent balance between speed and quality, requiring roughly 5GB of RAM.

Building the Backend

Project Structure

voice-assistant/
├── main.py
├── engines/
│   ├── __init__.py
│   ├── stt.py
│   ├── llm.py
│   └── tts.py
├── models/
│   ├── Meta-Llama-3-8B-Instruct.Q4_K_M.gguf
│   ├── ggml-base.en.bin
│   └── en_US-lessac-medium.onnx
└── requirements.txt

The LLM Engine

Start with the core component — the LLM engine that wraps llama-cpp-python:

# engines/llm.py
from llama_cpp import Llama
from typing import Generator, Optional
import os

MODEL_PATH = os.getenv(
    "LLM_MODEL_PATH",
    "models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"
)

class LLMEngine:
    def __init__(self):
        self.llm = Llama(
            model_path=MODEL_PATH,
            n_ctx=4096,
            n_threads=os.cpu_count(),
            n_gpu_layers=-1,  # Offload all layers to GPU if available
            verbose=False,
        )
        self.system_prompt = (
            "You are a helpful voice assistant. Keep responses concise "
            "and conversational. Avoid markdown formatting, code blocks, "
            "or special characters since responses will be spoken aloud. "
            "Aim for 1-3 sentences unless the user asks for detail."
        )

    def generate(self, user_input: str, history: list = None) -> str:
        messages = [{"role": "system", "content": self.system_prompt}]

        if history:
            for turn in history[-6:]:  # Keep last 6 turns for context
                messages.append({"role": "user", "content": turn["user"]})
                messages.append({"role": "assistant", "content": turn["assistant"]})

        messages.append({"role": "user", "content": user_input})

        response = self.llm.create_chat_completion(
            messages=messages,
            max_tokens=256,
            temperature=0.7,
            top_p=0.9,
            stop=["<|end_of_text|>", "<|eot_id|>"],
        )

        return response["choices"][0]["message"]["content"].strip()

    def generate_stream(
        self, user_input: str, history: list = None
    ) -> Generator[str, None, None]:
        messages = [{"role": "system", "content": self.system_prompt}]

        if history:
            for turn in history[-6:]:
                messages.append({"role": "user", "content": turn["user"]})
                messages.append({"role": "assistant", "content": turn["assistant"]})

        messages.append({"role": "user", "content": user_input})

        stream = self.llm.create_chat_completion(
            messages=messages,
            max_tokens=256,
            temperature=0.7,
            top_p=0.9,
            stream=True,
            stop=["<|end_of_text|>", "<|eot_id|>"],
        )

        for chunk in stream:
            delta = chunk["choices"][0].get("delta", {})
            if "content" in delta and delta["content"]:
                yield delta["content"]

The Speech-to-Text Engine

The STT engine uses whisper.cpp via subprocess calls. This approach avoids heavy Python dependencies and gives you direct access to the optimized C++ binary:

# engines/stt.py
import subprocess
import tempfile
import os
import wave

WHISPER_CPP_PATH = os.getenv("WHISPER_CPP_PATH", "whisper.cpp/main")
WHISPER_MODEL_PATH = os.getenv(
    "WHISPER_MODEL_PATH",
    "whisper.cpp/models/ggml-base.en.bin"
)

class STTEngine:
    def __init__(self):
        if not os.path.exists(WHISPER_CPP_PATH):
            raise FileNotFoundError(f"whisper.cpp binary not found at {WHISPER_CPP_PATH}")
        if not os.path.exists(WHISPER_MODEL_PATH):
            raise FileNotFoundError(f"Whisper model not found at {WHISPER_MODEL_PATH}")

    def transcribe(self, audio_path: str) -> str:
        """Transcribe a WAV file to text."""
        result = subprocess.run(
            [
                WHISPER_CPP_PATH,
                "-m", WHISPER_MODEL_PATH,
                "-f", audio_path,
                "-nt",  # No timestamps
                "-np",  # No progress
            ],
            capture_output=True,
            text=True,
            timeout=30,
        )

        if result.returncode != 0:
            raise RuntimeError(f"Whisper failed: {result.stderr}")

        # whisper.cpp outputs the transcription to stdout
        transcript = result.stdout.strip()
        return transcript

    def transcribe_bytes(self, audio_bytes: bytes, 
                         sample_rate: int = 16000) -> str:
        """Transcribe raw PCM audio bytes."""
        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
            temp_path = f.name

        try:
            with wave.open(temp_path, "wb") as wav_file:
                wav_file.setnchannels(1)
                wav_file.setsampwidth(2)  # 16-bit
                wav_file.setframerate(sample_rate)
                wav_file.writeframes(audio_bytes)

            return self.transcribe(temp_path)
        finally:
            os.unlink(temp_path)

The Text-to-Speech Engine

The TTS engine wraps piper to convert LLM responses back into audio:

# engines/tts.py
import subprocess
import tempfile
import os

PIPER_PATH = os.getenv("PIPER_PATH", "piper/piper")
PIPER_MODEL_PATH = os.getenv(
    "PIPER_MODEL_PATH",
    "models/en_US-lessac-medium.onnx"
)

class TTSEngine:
    def __init__(self):
        if not os.path.exists(PIPER_PATH):
            raise FileNotFoundError(f"Piper binary not found at {PIPER_PATH}")
        if not os.path.exists(PIPER_MODEL_PATH):
            raise FileNotFoundError(f"Piper model not found at {PIPER_MODEL_PATH}")

    def synthesize(self, text: str) -> bytes:
        """Convert text to WAV audio bytes."""
        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
            output_path = f.name

        try:
            result = subprocess.run(
                [
                    PIPER_PATH,
                    "-m", PIPER_MODEL_PATH,
                    "-f", output_path,
                    "--output-raw", "false",
                ],
                input=text,
                capture_output=True,
                text=True,
                timeout=15,
            )

            if result.returncode != 0:
                raise RuntimeError(f"Piper failed: {result.stderr}")

            with open(output_path, "rb") as audio_file:
                return audio_file.read()
        finally:
            if os.path.exists(output_path):
                os.unlink(output_path)

The FastAPI Application

Now wire everything together with a FastAPI server that exposes endpoints for the full voice assistant pipeline:

# main.py
import os
import tempfile
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import StreamingResponse, Response
from pydantic import BaseModel
from typing import Optional
import json
import io

from engines.llm import LLMEngine
from engines.stt import STTEngine
from engines.tts import TTSEngine

app = FastAPI(title="Voice Assistant Backend", version="1.0.0")

# Initialize engines at startup
llm_engine = LLMEngine()
stt_engine = STTEngine()
tts_engine = TTSEngine()

# Simple in-memory conversation store (use Redis in production)
conversations = {}

class TextInput(BaseModel):
    text: str
    conversation_id: Optional[str] = "default"

class TextResponse(BaseModel):
    response: str
    conversation_id: str

@app.get("/health")
async def health():
    return {"status": "healthy", "model": "llama-3-8b-instruct"}

@app.post("/chat", response_model=TextResponse)
async def chat(input_data: TextInput):
    """Text-only endpoint for testing the LLM."""
    history = conversations.get(input_data.conversation_id, [])

    try:
        response_text = llm_engine.generate(input_data.text, history)
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

    # Update conversation history
    history.append({"user": input_data.text, "assistant": response_text})
    conversations[input_data.conversation_id] = history

    return TextResponse(response=response_text, 
                        conversation_id=input_data.conversation_id)

@app.post("/voice")
async def voice_assistant(
    audio: UploadFile = File(...),
    conversation_id: str = "default"
):
    """Full voice pipeline: audio in, audio out."""
    # Validate file type
    if not audio.filename.endswith((".wav", ".mp3")):
        raise HTTPException(
            status_code=400, 
            detail="Only WAV or MP3 files are supported"
        )

    # Save uploaded audio temporarily
    with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
        content = await audio.read()
        f.write(content)
        temp_audio_path = f.name

    try:
        # Step 1: Speech to Text
        transcript = stt_engine.transcribe(temp_audio_path)

        if not transcript:
            transcript = "I didn't catch that. Could you repeat?"

        # Step 2: LLM generates response
        history = conversations.get(conversation_id, [])
        response_text = llm_engine.generate(transcript, history)

        # Update history
        history.append({"user": transcript, "assistant": response_text})
        conversations[conversation_id] = history

        # Step 3: Text to Speech
        audio_bytes = tts_engine.synthesize(response_text)

        return Response(
            content=audio_bytes,
            media_type="audio/wav",
            headers={
                "X-Transcript": transcript,
                "X-Response-Text": response_text,
            }
        )

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
    finally:
        os.unlink(temp_audio_path)

@app.post("/voice/stream")
async def voice_assistant_stream(
    audio: UploadFile = File(...),
    conversation_id: str = "default"
):
    """Streaming endpoint that returns JSON events."""
    with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
        content = await audio.read()
        f.write(content)
        temp_audio_path = f.name

    try:
        transcript = stt_engine.transcribe(temp_audio_path)
        history = conversations.get(conversation_id, [])

        def event_stream():
            # Send transcript event
            yield f"data: {json.dumps({'type': 'transcript', 'text': transcript})}\n\n"

            # Stream LLM tokens
            full_response = ""
            for token in llm_engine.generate_stream(transcript, history):
                full_response += token
                yield f"data: {json.dumps({'type': 'token', 'text': token})}\n\n"

            # Generate and send audio
            audio_bytes = tts_engine.synthesize(full_response)
            import base64
            audio_b64 = base64.b64encode(audio_bytes).decode()
            yield f"data: {json.dumps({'type': 'audio', 'data': audio_b64})}\n\n"

            yield f"data: {json.dumps({'type': 'done'})}\n\n"

            # Update history
            history.append({"user": transcript, "assistant": full_response})
            conversations[conversation_id] = history

        return StreamingResponse(
            event_stream(),
            media_type="text/event-stream"
        )

    finally:
        os.unlink(temp_audio_path)

@app.delete("/conversation/{conversation_id}")
async def clear_conversation(conversation_id: str):
    """Clear conversation history."""
    if conversation_id in conversations:
        del conversations[conversation_id]
    return {"status": "cleared", "conversation_id": conversation_id}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

Running the Server

Start the backend with:

export LLM_MODEL_PATH="models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"
export WHISPER_CPP_PATH="whisper.cpp/main"
export WHISPER_MODEL_PATH="whisper.cpp/models/ggml-base.en.bin"
export PIPER_PATH="piper/piper"
export PIPER_MODEL_PATH="models/en_US-lessac-medium.onnx"

python main.py

Test the text endpoint first to verify the LLM is working:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"text": "What is the capital of France?", "conversation_id": "test"}'

Then test the full voice pipeline with an audio file:

curl -X POST http://localhost:8000/voice \
  -F "audio=@test.wav" \
  -F "conversation_id=test" \
  -o response.wav

Best Practices

Model Selection

Choose your models based on your hardware and quality requirements. For voice assistants, conversational ability matters more than coding or reasoning benchmarks. Smaller models like Phi-3 Mini (3.8B) or Llama 3.2 3B offer faster response times on limited hardware. The Q4_K_M quantization level is the sweet spot for most deployments — it preserves quality while cutting memory usage by 75% compared to FP16.

Context Window Management

Voice assistants accumulate context quickly. Without management, the context window fills up and performance degrades. Implement a sliding window that keeps only the most recent turns, as shown in the LLM engine code above. For longer conversations, consider summarizing older turns periodically rather than dropping them entirely.

Latency Optimization

Latency is the enemy of voice assistants. Users expect responses within 1-2 seconds. Several strategies help:

• Use streaming inference so TTS can begin before the LLM finishes generating
• Pre-warm the model at startup with a dummy inference call
• Use smaller Whisper models (tiny or base) for STT — accuracy is sufficient for most voice commands
• Keep the LLM's max_tokens low (128-256) since spoken responses should be concise
• Pin the server to specific CPU cores using taskset to avoid context switching

System Prompt Engineering

The system prompt is your primary tool for shaping assistant behavior. For voice specifically, instruct the model to avoid markdown, code blocks, URLs, and special characters. These don't translate well to speech. Also instruct it to keep responses short — long monologues make for poor voice experiences.

Error Handling and Fallbacks

Every component can fail. Audio might be too noisy for Whisper, the LLM might produce empty output, or Piper might fail on unusual characters. Wrap each stage in try-catch blocks and provide graceful fallbacks. If STT fails, respond with "I didn't catch that." If the LLM produces empty output, retry with a simpler prompt. If TTS fails on special characters, strip them and retry.

Security Considerations

Even though everything runs locally, security still matters. Validate uploaded file types and sizes. Rate-limit requests to prevent resource exhaustion. If you expose the API externally, add authentication. The in-memory conversation store in this tutorial should be replaced with Redis or a database for production use, with appropriate access controls.

Monitoring and Logging

Log the latency of each pipeline stage separately. This helps identify bottlenecks — if STT takes 3 seconds but the LLM takes 0.5 seconds, you know where to optimize. Track token counts, model load times, and error rates. Tools like Prometheus and Grafana work well for visualizing these metrics over time.

Conclusion

Building a voice assistant backend with llama.cpp gives you a fully local, private, and cost-effective alternative to cloud APIs. The architecture is straightforward — STT feeds into the LLM, which feeds into TTS — but the real value lies in the control you gain over every component. You can swap models, tune prompts, adjust quantization levels, and optimize latency without depending on external services. The code in this tutorial provides a solid foundation; from here, you can add wake word detection, multi-speaker support, tool calling, or integration with smart home systems. The entire stack runs on a single machine, scales with your hardware, and keeps every byte of user data under your control.