Introduction to Offline AI Assistants
An offline AI assistant is a conversational AI system that runs entirely on your local machine, without requiring an internet connection or sending data to remote servers. These assistants leverage open-source language models that have been optimized to run on consumer hardware, giving you the power of AI while maintaining complete privacy and control over your data.
Thanks to recent advances in model quantization and inference optimization, what once required massive data center GPUs can now run on a modern laptop. Tools like Ollama, llama.cpp, and LM Studio have made local AI accessible to developers of all skill levels.
Why Local Models Matter
Building an AI assistant with local models offers several compelling advantages over cloud-based alternatives:
- Privacy: Your conversations never leave your device. This is critical for handling sensitive information in healthcare, legal, or personal contexts.
- Cost: No API fees or subscription costs. Once set up, you can run unlimited queries at no additional expense.
- Offline capability: Works without an internet connection, making it ideal for travel, remote locations, or air-gapped environments.
- Low latency: No network round-trips means faster response times for many use cases.
- Customization: Full control over the model, system prompts, and fine-tuning without vendor restrictions.
- Data sovereignty: Compliance with regulations like GDPR or HIPAA is easier when data stays local.
Prerequisites and Setup
Before building your offline AI assistant, you'll need the following:
- A machine with at least 8GB RAM (16GB or more recommended for larger models)
- Python 3.9 or later installed
- Basic familiarity with Python and command-line tools
- Optional but helpful: A GPU (NVIDIA with CUDA or Apple Silicon) for faster inference
Installing Ollama
Ollama is one of the easiest ways to run local models. It handles model downloading, quantization, and serving through a simple API.
On macOS or Linux, install with:
curl -fsSL https://ollama.com/install.sh | shOn Windows, download the installer from ollama.com and run it.
Once installed, pull a model. For this tutorial, we'll use Llama 3.2, which offers a good balance of performance and capability:
ollama pull llama3.2Verify the installation by running:
ollama run llama3.2 "Hello, are you working?"Building the Core Assistant
Setting Up the Python Environment
Create a new project directory and set up a virtual environment:
mkdir offline-assistant
cd offline-assistant
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install ollama richThe ollama package provides a Python client for the Ollama API, and rich will give us nice terminal formatting.
Creating the Basic Assistant
Create a file called assistant.py and add the following code:
import ollama
from rich.console import Console
from rich.markdown import Markdown
console = Console()
SYSTEM_PROMPT = """You are a helpful, knowledgeable AI assistant running locally.
You provide clear, accurate, and concise responses.
When you don't know something, you say so honestly."""
def chat_with_assistant():
console.print("[bold green]Offline AI Assistant[/bold green]")
console.print("[dim]Type 'quit' to exit, 'clear' to reset conversation[/dim]\n")
messages = [
{"role": "system", "content": SYSTEM_PROMPT}
]
while True:
try:
user_input = console.input("[bold blue]You:[/bold blue] ")
if user_input.lower() == "quit":
console.print("[yellow]Goodbye![/yellow]")
break
if user_input.lower() == "clear":
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
console.print("[dim]Conversation cleared.[/dim]\n")
continue
if not user_input.strip():
continue
messages.append({"role": "user", "content": user_input})
console.print("[bold green]Assistant:[/bold green]")
response = ollama.chat(
model="llama3.2",
messages=messages,
stream=True
)
full_response = ""
for chunk in response:
content = chunk["message"]["content"]
full_response += content
console.print(content, end="")
console.print("\n")
messages.append({"role": "assistant", "content": full_response})
except KeyboardInterrupt:
console.print("\n[yellow]Goodbye![/yellow]")
break
except Exception as e:
console.print(f"\n[red]Error: {e}[/red]\n")
if __name__ == "__main__":
chat_with_assistant()
Run the assistant with:
python assistant.pyYou now have a working offline AI assistant! The streaming response gives you a ChatGPT-like experience, and the conversation history is maintained so the assistant remembers context.
Adding Conversation Persistence
To save conversations between sessions, let's add JSON-based persistence. Create a new file called persistent_assistant.py:
import ollama
import json
import os
from datetime import datetime
from rich.console import Console
from rich.markdown import Markdown
from rich.panel import Panel
console = Console()
CONVERSATION_FILE = "conversations.json"
SYSTEM_PROMPT = """You are a helpful, knowledgeable AI assistant running locally.
You provide clear, accurate, and concise responses."""
def load_conversations():
if os.path.exists(CONVERSATION_FILE):
with open(CONVERSATION_FILE, "r") as f:
return json.load(f)
return {}
def save_conversations(conversations):
with open(CONVERSATION_FILE, "w") as f:
json.dump(conversations, f, indent=2)
def list_conversations(conversations):
if not conversations:
console.print("[dim]No saved conversations.[/dim]")
return None
console.print("[bold]Saved Conversations:[/bold]")
for i, conv_id in enumerate(conversations.keys(), 1):
conv = conversations[conv_id]
first_msg = conv["messages"][1]["content"][:50] if len(conv["messages"]) > 1 else "Empty"
console.print(f" {i}. [{conv_id}] {first_msg}...")
choice = console.input("\nSelect conversation number (or 'new'): ")
if choice.lower() == "new":
return None
try:
conv_id = list(conversations.keys())[int(choice) - 1]
return conv_id
except (ValueError, IndexError):
console.print("[red]Invalid selection.[/red]")
return None
def chat_with_persistence():
conversations = load_conversations()
console.print(Panel.fit(
"[bold green]Offline AI Assistant with Persistence[/bold green]\n"
"[dim]Commands: 'quit', 'clear', 'list', 'new'[/dim]",
border_style="green"
))
conv_id = datetime.now().strftime("%Y%m%d_%H%M%S")
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
if conversations:
selected = list_conversations(conversations)
if selected:
conv_id = selected
messages = conversations[conv_id]["messages"]
console.print(f"[dim]Resumed conversation: {conv_id}[/dim]\n")
while True:
try:
user_input = console.input("[bold blue]You:[/bold blue] ")
if user_input.lower() == "quit":
conversations[conv_id] = {
"timestamp": datetime.now().isoformat(),
"messages": messages
}
save_conversations(conversations)
console.print("[yellow]Conversation saved. Goodbye![/yellow]")
break
if user_input.lower() == "clear":
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
console.print("[dim]Conversation cleared.[/dim]\n")
continue
if user_input.lower() == "list":
selected = list_conversations(conversations)
if selected:
conv_id = selected
messages = conversations[conv_id]["messages"]
console.print(f"[dim]Switched to: {conv_id}[/dim]\n")
continue
if user_input.lower() == "new":
conv_id = datetime.now().strftime("%Y%m%d_%H%M%S")
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
console.print(f"[dim]New conversation: {conv_id}[/dim]\n")
continue
if not user_input.strip():
continue
messages.append({"role": "user", "content": user_input})
console.print("[bold green]Assistant:[/bold green]")
response = ollama.chat(
model="llama3.2",
messages=messages,
stream=True
)
full_response = ""
for chunk in response:
content = chunk["message"]["content"]
full_response += content
console.print(content, end="")
console.print("\n")
messages.append({"role": "assistant", "content": full_response})
conversations[conv_id] = {
"timestamp": datetime.now().isoformat(),
"messages": messages
}
save_conversations(conversations)
except KeyboardInterrupt:
conversations[conv_id] = {
"timestamp": datetime.now().isoformat(),
"messages": messages
}
save_conversations(conversations)
console.print("\n[yellow]Saved. Goodbye![/yellow]")
break
except Exception as e:
console.print(f"\n[red]Error: {e}[/red]\n")
if __name__ == "__main__":
chat_with_persistence()
Adding Document Knowledge with RAG
A truly useful assistant should be able to reference your own documents. Retrieval-Augmented Generation (RAG) allows the model to search through your files and use them as context. Let's build this capability using local embeddings.
Installing RAG Dependencies
pip install chromadb sentence-transformers pypdfBuilding the RAG Assistant
Create a file called rag_assistant.py:
import ollama
import chromadb
import os
import glob
from sentence_transformers import SentenceTransformer
from rich.console import Console
from rich.panel import Panel
console = Console()
SYSTEM_PROMPT = """You are a helpful AI assistant with access to a local document knowledge base.
Use the provided context to answer questions accurately.
If the context doesn't contain relevant information, say so and answer from your general knowledge.
Always cite which document your information comes from."""
class DocumentStore:
def __init__(self, db_path="./chroma_db"):
self.client = chromadb.PersistentClient(path=db_path)
self.collection = self.client.get_or_create_collection("documents")
self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
def add_document(self, text, source, chunk_size=500, overlap=50):
"""Add a document to the store, splitting it into chunks."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end]
chunks.append(chunk)
start = end - overlap
for i, chunk in enumerate(chunks):
embedding = self.embedder.encode(chunk).tolist()
doc_id = f"{source}_chunk_{i}"
self.collection.add(
ids=[doc_id],
embeddings=[embedding],
documents=[chunk],
metadatas=[{"source": source, "chunk": i}]
)
console.print(f"[green]Added {len(chunks)} chunks from {source}[/green]")
def add_pdf(self, file_path):
"""Extract text from a PDF and add it to the store."""
from pypdf import PdfReader
reader = PdfReader(file_path)
text = ""
for page in reader.pages:
text += page.extract_text() + "\n"
self.add_document(text, os.path.basename(file_path))
def add_text_file(self, file_path):
"""Add a plain text file to the store."""
with open(file_path, "r", encoding="utf-8") as f:
text = f.read()
self.add_document(text, os.path.basename(file_path))
def add_directory(self, dir_path):
"""Add all supported files from a directory."""
extensions = ["*.txt", "*.md", "*.pdf"]
for ext in extensions:
for file_path in glob.glob(os.path.join(dir_path, "**", ext), recursive=True):
console.print(f"[dim]Processing: {file_path}[/dim]")
if ext == "*.pdf":
self.add_pdf(file_path)
else:
self.add_text_file(file_path)
def search(self, query, n_results=3):
"""Search for relevant document chunks."""
query_embedding = self.embedder.encode(query).tolist()
results = self.collection.query(
query_embeddings=[query_embedding],
n_results=n_results
)
context_parts = []
for i, (doc, metadata) in enumerate(zip(results["documents"][0], results["metadatas"][0])):
context_parts.append(f"[Document: {metadata['source']}, Chunk: {metadata['chunk']}]\n{doc}")
return "\n\n---\n\n".join(context_parts)
def rag_chat():
store = DocumentStore()
console.print(Panel.fit(
"[bold green]Offline RAG Assistant[/bold green]\n"
"[dim]Commands: 'add <path>', 'add-dir <path>', 'quit'[/dim]",
border_style="green"
))
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
while True:
try:
user_input = console.input("[bold blue]You:[/bold blue] ")
if user_input.lower() == "quit":
break
if user_input.lower().startswith("add "):
path = user_input[4:].strip()
if os.path.isfile(path):
if path.endswith(".pdf"):
store.add_pdf(path)
else:
store.add_text_file(path)
else:
console.print(f"[red]File not found: {path}[/red]")
continue
if user_input.lower().startswith("add-dir "):
path = user_input[8:].strip()
if os.path.isdir(path):
store.add_directory(path)
else:
console.print(f"[red]Directory not found: {path}[/red]")
continue
if not user_input.strip():
continue
# Retrieve relevant context
context = store.search(user_input, n_results=3)
# Build augmented prompt
augmented_prompt = f"""Context from your document knowledge base:
{context}
---
User Question: {user_input}
Please answer the question using the context above when relevant."""
messages.append({"role": "user", "content": augmented_prompt})
console.print("[bold green]Assistant:[/bold green]")
response = ollama.chat(
model="llama3.2",
messages=messages,
stream=True
)
full_response = ""
for chunk in response:
content = chunk["message"]["content"]
full_response += content
console.print(content, end="")
console.print("\n")
messages.append({"role": "assistant", "content": full_response})
except KeyboardInterrupt:
console.print("\n[yellow]Goodbye![/yellow]")
break
except Exception as e:
console.print(f"\n[red]Error: {e}[/red]\n")
if __name__ == "__main__":
rag_chat()
With this setup, you can index your own documents and ask questions about them:
python rag_assistant.py
> add ./my_notes.txt
> add ./research_paper.pdf
> add-dir ./documents/
> What does my notes file say about project deadlines?Building a Web Interface
For a more polished experience, let's create a simple web interface using Gradio:
pip install gradioCreate web_assistant.py:
import ollama
import gradio as gr
SYSTEM_PROMPT = """You are a helpful, knowledgeable AI assistant running locally.
Provide clear, accurate, and well-structured responses."""
def respond(message, history):
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
for human, assistant in history:
messages.append({"role": "user", "content": human})
messages.append({"role": "assistant", "content": assistant})
messages.append({"role": "user", "content": message})
response = ollama.chat(
model="llama3.2",
messages=messages,
stream=True
)
partial_response = ""
for chunk in response:
partial_response += chunk["message"]["content"]
yield partial_response
demo = gr.ChatInterface(
fn=respond,
title="Offline AI Assistant",
description="Running entirely on your local machine with Llama 3.2",
theme=gr.themes.Soft(),
retry_btn="Retry",
undo_btn="Undo",
clear_btn="Clear",
)
if __name__ == "__main__":
demo.launch(server_name="127.0.0.1", server_port=7860)
Run it and open your browser to http://127.0.0.1:7860:
python web_assistant.pyAdding Tool Use and Function Calling
Modern local models can call functions, enabling your assistant to perform actions. Here's how to add tool use:
import ollama
import json
import subprocess
import os
from datetime import datetime
from rich.console import Console
console = Console()
TOOLS = [
{
"type": "function",
"function": {
"name": "run_shell_command",
"description": "Run a shell command on the local system",
"parameters": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "The shell command to execute"
}
},
"required": ["command"]
}
}
},
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read the contents of a file",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Path to the file to read"
}
},
"required": ["path"]
}
}
},
{
"type": "function",
"function": {
"name": "write_file",
"description": "Write content to a file",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Path to the file to write"
},
"content": {
"type": "string",
"description": "Content to write to the file"
}
},
"required": ["path", "content"]
}
}
},
{
"type": "function",
"function": {
"name": "get_current_time",
"description": "Get the current date and time",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
}
]
def execute_tool(name, args):
"""Execute a tool and return the result."""
try:
if name == "run_shell_command":
result = subprocess.run(
args["command"],
shell=True,
capture_output=True,
text=True,
timeout=30
)
output = result.stdout
if result.stderr:
output += f"\nSTDERR: {result.stderr}"
return output if output else "Command completed with no output."
elif name == "read_file":
with open(args["path"], "r") as f:
return f.read()
elif name == "write_file":
with open(args["path"], "w") as f:
f.write(args["content"])
return f"Successfully wrote to {args['path']}"
elif name == "get_current_time":
return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
else:
return f"Unknown tool: {name}"
except Exception as e:
return f"Error executing {name}: {str(e)}"
def chat_with_tools():
console.print("[bold green]Offline AI Assistant with Tools[/bold green]")
console.print("[dim]The assistant can run commands, read/write files, and check the time.[/dim]\n")
messages = [
{"role": "system", "content": "You are a helpful assistant with access to tools. Use them when appropriate to help the user. Always explain what you're doing before using a tool."}
]
while True:
try:
user_input = console.input("[bold blue]You:[/bold blue] ")
if user_input.lower() == "quit":
break
if not user_input.strip():
continue
messages.append({"role": "user", "content": user_input})
# Keep calling the model until it stops requesting tools
while True:
response = ollama.chat(
model="llama3.2",
messages=messages,
tools=TOOLS
)
assistant_message = response["message"]
messages.append(assistant_message)
# Check if the model wants to use tools
if not assistant_message.get("tool_calls"):
console.print(f"[bold green]Assistant:[/bold green] {assistant_message['content']}\n")
break
# Execute each tool call
for tool_call in assistant_message["tool_calls"]:
func = tool_call["function"]
console.print(f"[yellow]Calling tool: {func['name']}[/yellow]")
console.print(f"[dim]Arguments: {json.dumps(func.get('arguments', {}))}[/dim]")
result = execute_tool(func["name"], func.get("arguments", {}))
console.print(f"[dim]Result: {result[:200]}...[/dim]\n")
messages.append({
"role": "tool",
"content": result
})
except KeyboardInterrupt:
console.print("\n[yellow]Goodbye![/yellow]")
break
except Exception as e:
console.print(f"\n[red]Error: {e}[/red]\n")
if __name__ == "__main__":
chat_with_tools()
Choosing the Right Model
Selecting the right model depends on your hardware and use case. Here are some popular options:
- Llama 3.2 (3B): Excellent for general chat on modest hardware. Requires about 2GB RAM.
- Llama 3.1 (8B): Better reasoning and knowledge. Needs 5-6GB RAM.
- Mistral (7B): Strong performance for code and general tasks. Similar requirements to Llama 3.1 8B.
- Phi-3 Mini (3.8B): Microsoft's compact model with impressive reasoning for its size.
- Qwen 2.5 (7B): Excellent multilingual support and coding capabilities.
- CodeLlama (7B-13B): Specialized for code generation and understanding.
- Llama 3.1 (70B): High-end model requiring 40GB+ RAM but approaching GPT-4 quality.
To switch models, simply pull a new one and update your code:
ollama pull mistral
# Then change the model parameter in your code:
# model="mistral" instead of model="llama3.2"Performance Optimization
Context Window Management
Local models have limited context windows. Managing conversation history efficiently prevents slowdowns:
def trim_messages(messages, max_messages=20):
"""Keep system prompt and last N messages."""
system = [m for m in messages if m["role"] == "system"]
conversation = [m for m in messages if m["role"] != "system"]
if len(conversation) > max_messages:
conversation = conversation[-max_messages:]
return system + conversation
# Use in your chat loop:
messages = trim_messages(messages, max_messages=20)Model Parameters
Fine-tune inference parameters for better responses:
response = ollama.chat(
model="llama3.2",
messages=messages,
options={
"temperature": 0.7, # Creativity (0=deterministic, 1=very random)
"top_p": 0.9, # Nucleus sampling
"top_k": 40, # Top-k sampling
"num_ctx": 4096, # Context window size
"num_predict": 512, # Max tokens to generate
"repeat_penalty": 1.1, # Penalize repetition
"seed": 42 # For reproducible outputs
},
stream=True
)GPU Acceleration
Ollama automatically detects and uses available GPUs. To check if GPU acceleration is active:
ollama psThe output will show which processor is being used. If you have a GPU but it's not being used, ensure your drivers are up to date.
Best Practices
- Start small: Begin with a 3B model to verify everything works, then scale up based on your hardware.
- Monitor resource usage: Watch RAM and VRAM consumption. Running out of memory causes severe slowdowns as the system swaps to disk.
- Use appropriate quantization: Ollama uses 4-bit quantization by default, which offers a good speed/quality balance. You can pull different quantization levels if needed.
- Craft good system prompts: The system prompt shapes the assistant's behavior significantly. Be specific about tone, format, and constraints.
- Implement error handling: Local models can occasionally produce malformed outputs. Always handle exceptions gracefully.
- Validate tool outputs: When using function calling, validate and sanitize inputs before executing commands, especially shell commands.
- Keep models updated: New model versions often bring significant improvements. Periodically check for updates with
ollama pull. - Test with your use case: Different models excel at different tasks. Benchmark several models on your specific workload before committing.
- Consider model switching: Use smaller models for simple queries and larger models for complex reasoning. You can route based on query complexity.
- Secure your setup: If using tool use, run in a container or restricted environment to prevent unintended system access.
Conclusion
Building an offline AI assistant with local models is now practical and straightforward thanks to tools like Ollama and the growing ecosystem of open-source models. You get the power of conversational AI with complete privacy, zero ongoing costs, and full control over the system. Starting with the basic chat interface, you can progressively add features like conversation persistence, document retrieval with RAG, web interfaces, and tool use to create a capable assistant tailored to your needs. As open-source models continue to improve at a rapid pace, local AI assistants will only become more powerful, making now an excellent time to start building. Whether you need a private coding companion, a document analysis tool, or a general-purpose assistant that works anywhere, the building blocks are in your hands.