Building a Local Voice Assistant with Whisper and Ollama

Building a Local Voice Assistant with Whisper and Ollama

Voice assistants have become a staple of modern computing, but most popular solutions—Siri, Alexa, Google Assistant—rely on cloud infrastructure. Every spoken word is shipped to remote servers for processing, raising legitimate concerns about privacy, latency, cost, and offline availability. In this tutorial, you'll learn how to build a fully local voice assistant that transcribes speech with OpenAI's Whisper and generates intelligent responses with Ollama, a local large language model runtime. The entire pipeline runs on your machine, with no data ever leaving your network.

What You're Building

The assistant follows a simple but powerful loop: capture audio from your microphone, transcribe it to text with Whisper, send that text to an Ollama-hosted LLM, and speak the response back using a text-to-speech engine. The architecture is modular, so each component can be swapped or upgraded independently.

The core pipeline looks like this:

Microphone Audio → Whisper (STT) → Text Prompt → Ollama (LLM) → Response Text → TTS → Speakers

Why Local Matters

Running a voice assistant locally offers several concrete advantages. Privacy is the most obvious: sensitive conversations, business discussions, or personal queries never touch third-party servers. Latency improves dramatically when you eliminate network round-trips, especially for short interactions. Cost disappears entirely—no per-token API fees, no subscription tiers. Finally, offline operation becomes possible, which matters for field work, secure environments, or unreliable connections.

The tradeoff is hardware. Whisper's smaller models run comfortably on modern CPUs, while Ollama's quantized LLMs need roughly 8GB of RAM for a 7B parameter model and a GPU helps significantly for real-time interaction. We'll address optimization strategies later in the tutorial.

Prerequisites and Installation

Before writing code, install the required tools. You'll need Python 3.10 or newer, FFmpeg for audio handling, and Ollama itself.

Installing Ollama

Ollama is available for macOS, Linux, and Windows. Download the installer from the official site or use the install script on Linux:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, pull a model. For a voice assistant, the llama3.2 (3B parameters) or llama3.1 (8B parameters) models work well. The smaller model is faster and more suitable for real-time conversation:

ollama pull llama3.2

Verify the model is available:

ollama list

Installing Python Dependencies

Create a virtual environment and install the required packages:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install openai-whisper ollama pyttsx3 sounddevice numpy scipy

Here's what each package does:

• openai-whisper — OpenAI's speech recognition model, runs locally
• ollama — Python client for the local Ollama server
• pyttsx3 — Offline text-to-speech engine
• sounddevice — Captures microphone input
• numpy and scipy — Audio array manipulation and file I/O

You'll also need FFmpeg installed system-wide, which Whisper uses internally. On macOS use brew install ffmpeg, on Ubuntu sudo apt install ffmpeg, and on Windows download the binary and add it to your PATH.

Building the Speech-to-Text Component

Whisper is the foundation of our input pipeline. It supports multiple model sizes, from tiny (39M parameters, fastest, least accurate) to large-v3 (1.5B parameters, most accurate). For a voice assistant, base or small offers a good balance between speed and accuracy on CPU. If you have a GPU, medium becomes viable.

Create a file named stt.py:

import whisper
import sounddevice as sd
import numpy as np
from scipy.io.wavfile import write

class SpeechToText:
    def __init__(self, model_size="base"):
        print(f"Loading Whisper model '{model_size}'...")
        self.model = whisper.load_model(model_size)
        print("Whisper model loaded.")

    def record_audio(self, duration=5, sample_rate=16000):
        print(f"Recording for {duration} seconds...")
        audio = sd.rec(int(duration * sample_rate),
                       samplerate=sample_rate,
                       channels=1,
                       dtype="float32")
        sd.wait()
        print("Recording complete.")
        # Whisper expects int16 PCM
        audio_int16 = (audio * 32767).astype(np.int16)
        return audio_int16.flatten(), sample_rate

    def transcribe(self, audio_data, sample_rate=16000):
        # Write to a temporary in-memory approach using numpy directly
        # Whisper's transcribe accepts a numpy array
        result = self.model.transcribe(audio_data.astype(np.float32),
                                       fp16=False,
                                       language="en")
        return result["text"].strip()

    def listen(self, duration=5):
        audio, sr = self.record_audio(duration)
        text = self.transcribe(audio, sr)
        return text

The record_audio method captures mono audio at 16kHz, which is Whisper's expected sample rate. The transcribe method accepts a numpy array directly, avoiding the overhead of writing temporary WAV files. Setting fp16=False is important when running on CPU; on GPU you can leave it as default for a speed boost.

Connecting to Ollama for LLM Responses

Ollama runs as a local server on port 11434 by default. The Python client communicates with it over HTTP. We'll wrap the client in a class that maintains conversation context, which is essential for a natural voice assistant that can handle follow-up questions.

Create llm.py:

import ollama

class LanguageModel:
    def __init__(self, model_name="llama3.2", system_prompt=None):
        self.model_name = model_name
        self.system_prompt = system_prompt or (
            "You are a helpful voice assistant. Keep responses concise "
            "and conversational, ideally under three sentences. Avoid "
            "markdown formatting, code blocks, or special characters "
            "since responses will be spoken aloud."
        )
        self.messages = [{"role": "system", "content": self.system_prompt}]

    def chat(self, user_input):
        self.messages.append({"role": "user", "content": user_input})
        response = ollama.chat(
            model=self.model_name,
            messages=self.messages
        )
        reply = response["message"]["content"]
        self.messages.append({"role": "assistant", "content": reply})
        return reply

    def reset(self):
        self.messages = [{"role": "system", "content": self.system_prompt}]

The system prompt is deliberately tuned for voice output. LLMs tend to produce verbose, formatted text that sounds unnatural when read aloud. Instructing the model to be concise and avoid special characters dramatically improves the spoken experience.

Adding Text-to-Speech Output

For TTS, pyttsx3 works offline and requires no external services. It uses your operating system's built-in speech engines—SAPI5 on Windows, NSSpeechSynthesizer on macOS, and eSpeak on Linux.

Create tts.py:

import pyttsx3

class TextToSpeech:
    def __init__(self, rate=180, voice_index=None):
        self.engine = pyttsx3.init()
        self.engine.setProperty("rate", rate)
        voices = self.engine.getProperty("voices")
        if voice_index is not None and voice_index < len(voices):
            self.engine.setProperty("voice", voices[voice_index].id)

    def speak(self, text):
        self.engine.say(text)
        self.engine.runAndWait()

    def save_to_file(self, text, filename):
        self.engine.save_to_file(text, filename)
        self.engine.runAndWait()

The rate property controls speech speed. A value of 180 words per minute is a reasonable default; adjust based on your preference. The voice_index parameter lets you select between available system voices, which is useful if you want a specific gender or accent.

Assembling the Complete Assistant

Now combine all three components into a single runnable script. Create assistant.py:

from stt import SpeechToText
from llm import LanguageModel
from tts import TextToSpeech

class VoiceAssistant:
    def __init__(self, whisper_model="base", llm_model="llama3.2"):
        self.stt = SpeechToText(model_size=whisper_model)
        self.llm = LanguageModel(model_name=llm_model)
        self.tts = TextToSpeech(rate=180)
        self.listening_duration = 5

    def run(self):
        print("=" * 50)
        print("  Local Voice Assistant Ready")
        print("  Press Ctrl+C to exit")
        print("=" * 50)
        self.tts.speak("Voice assistant ready. How can I help you?")

        try:
            while True:
                print("\nListening...")
                user_text = self.stt.listen(duration=self.listening_duration)

                if not user_text or len(user_text.strip()) < 2:
                    print("Didn't catch that. Try again.")
                    continue

                print(f"You said: {user_text}")

                print("Thinking...")
                response = self.llm.chat(user_text)
                print(f"Assistant: {response}")

                self.tts.speak(response)

        except KeyboardInterrupt:
            print("\nShutting down. Goodbye!")
            self.tts.speak("Goodbye!")


if __name__ == "__main__":
    assistant = VoiceAssistant(
        whisper_model="base",
        llm_model="llama3.2"
    )
    assistant.run()

Run the assistant with:

python assistant.py

Speak into your microphone during the five-second recording window, and the assistant will transcribe, generate a response, and speak it back. The conversation context persists across exchanges, so you can ask follow-up questions naturally.

Adding Voice Activity Detection

Fixed-duration recording is functional but awkward—you must wait the full duration even if you finish speaking quickly, and long pauses get cut off. Voice Activity Detection (VAD) solves this by monitoring audio levels and stopping recording when silence is detected.

Here's an enhanced recording method using energy-based VAD:

import webrtcvad
import collections

class SpeechToText:
    def __init__(self, model_size="base", aggressiveness=3):
        self.model = whisper.load_model(model_size)
        self.vad = webrtcvad.Vad(aggressiveness)
        self.sample_rate = 16000
        self.frame_duration = 30  # ms
        self.frame_size = int(self.sample_rate * self.frame_duration / 1000)

    def record_with_vad(self, max_duration=10, silence_threshold=15):
        print("Listening (speak now)...")
        frames = []
        silent_frames = 0
        total_frames = 0
        max_frames = int(max_duration * 1000 / self.frame_duration)

        with sd.InputStream(samplerate=self.sample_rate,
                            channels=1,
                            dtype="int16",
                            blocksize=self.frame_size):
            pass  # Initialize stream

        stream = sd.InputStream(samplerate=self.sample_rate,
                                channels=1,
                                dtype="int16",
                                blocksize=self.frame_size)
        stream.start()

        try:
            while total_frames < max_frames:
                frame, overflowed = stream.read(self.frame_size)
                frame_bytes = frame.tobytes()
                is_speech = self.vad.is_speech(frame_bytes, self.sample_rate)

                if is_speech:
                    frames.append(frame.flatten())
                    silent_frames = 0
                else:
                    if len(frames) > 0:
                        silent_frames += 1
                    # Keep a few silent frames for natural pacing

                total_frames += 1

                if silent_frames > silence_threshold and len(frames) > 0:
                    break
        finally:
            stream.stop()
            stream.close()

        if not frames:
            return None, self.sample_rate

        audio = np.concatenate(frames)
        print("Recording complete.")
        return audio, self.sample_rate

Install the VAD dependency with pip install webrtcvad. The aggressiveness parameter ranges from 0 (most permissive) to 3 (most aggressive filtering). A value of 3 works well for quiet environments; lower it if the assistant cuts off soft speech.

Streaming Responses for Lower Latency

One limitation of the current design is that the assistant waits for the complete LLM response before speaking. For longer responses, this creates an uncomfortable delay. Ollama supports streaming, so you can begin TTS as soon as the first sentence arrives.

import re

class LanguageModel:
    def chat_stream(self, user_input):
        self.messages.append({"role": "user", "content": user_input})
        buffer = ""
        full_response = ""

        stream = ollama.chat(
            model=self.model_name,
            messages=self.messages,
            stream=True
        )

        for chunk in stream:
            token = chunk["message"]["content"]
            buffer += token
            full_response += token

            # Speak in sentence-sized chunks
            sentences = re.split(r'(?<=[.!?])\s+', buffer)
            if len(sentences) > 1:
                for sentence in sentences[:-1]:
                    yield sentence.strip()
                buffer = sentences[-1]

        if buffer.strip():
            yield buffer.strip()

        self.messages.append({"role": "assistant", "content": full_response})

Update the main loop to consume the generator:

print("Thinking...")
for sentence in self.llm.chat_stream(user_text):
    print(f"Assistant: {sentence}")
    self.tts.speak(sentence)

This approach starts speaking the first sentence while the LLM is still generating the rest, reducing perceived latency significantly.

Best Practices

• Choose the right Whisper model. Start with base for development. Move to small if accuracy matters more than speed, or tiny if you need maximum responsiveness on weak hardware.
• Use a GPU when available. Whisper and Ollama both benefit enormously from CUDA or Metal acceleration. Whisper loads onto GPU automatically if PyTorch detects one. For Ollama, GPU support is automatic on supported hardware.
• Tune the system prompt carefully. The LLM's output format directly affects speech quality. Explicitly forbid markdown, emojis, and long responses. Test with real conversations and iterate.
• Implement conversation memory limits. The messages list grows indefinitely. Trim older messages periodically to avoid context window overflow and slowing responses:

def trim_history(self, max_messages=20):
    if len(self.messages) > max_messages:
        self.messages = [self.messages[0]] + self.messages[-max_messages:]

• Handle errors gracefully. Microphone access can fail, Ollama might not be running, and Whisper can produce empty transcriptions. Wrap each stage in try-except blocks and provide spoken feedback when something goes wrong.
• Consider wake word detection. Instead of always listening, add a wake word using a lightweight model like Porcupine or a simple keyword match on Whisper's tiny model. This reduces resource usage and prevents accidental activations.
• Test with diverse audio conditions. Background noise, accents, and varying microphone distances all affect Whisper accuracy. Test in your actual deployment environment, not just quiet development settings.
• Explore alternative TTS engines. While pyttsx3 is convenient, its voice quality is limited. For better output, consider Piper (neural TTS that runs locally) or Coqui TTS. Both produce more natural-sounding speech while remaining fully offline.

Extending the Assistant

Once the basic pipeline works, you can extend it with function calling. Ollama supports tool use, allowing the assistant to execute Python functions based on user intent. For example, you could add a weather lookup, calendar integration, or smart home control:

import json
import datetime

def get_time():
    return datetime.datetime.now().strftime("%I:%M %p")

def get_date():
    return datetime.datetime.now().strftime("%A, %B %d, %Y")

AVAILABLE_TOOLS = {
    "get_time": get_time,
    "get_date": get_date,
}

def handle_tool_call(response_text):
    # Simple keyword-based tool routing
    text = response_text.lower()
    if "time" in text and "what" in text:
        return get_time()
    if "date" in text and "what" in text:
        return get_date()
    return None

For more robust tool use, leverage Ollama's native function calling support by defining tools in the chat request and parsing the structured output. This enables the LLM to decide when and which tool to invoke based on the user's intent.

Conclusion

Building a local voice assistant with Whisper and Ollama demonstrates that capable AI tooling no longer requires cloud dependencies. By combining Whisper's robust speech recognition, Ollama's efficient local LLM inference, and an offline TTS engine, you've created a complete conversational system that respects user privacy, works without internet access, and costs nothing to run. The modular architecture means each component can be upgraded independently—swap in a larger Whisper model for better accuracy, try different Ollama models for varied personalities, or replace the TTS engine with a neural voice for more natural output. As local AI models continue to improve, this pattern will only become more powerful, making private, offline voice assistants a practical reality for any developer.