Building a Private ChatGPT Alternative with Local Models

Building a Private ChatGPT Alternative with Local Models

Large language models have transformed how developers build applications, but relying on cloud-hosted APIs like OpenAI's ChatGPT introduces concerns around privacy, latency, cost, and vendor lock-in. Running models locally gives you full control over your data, predictable costs, and the ability to customize behavior without sending sensitive information to third-party servers. This tutorial walks through building a fully functional, private ChatGPT alternative using open-source models and tooling.

What It Is

A private ChatGPT alternative is a chat application powered by an open-weight language model that runs entirely on your own hardware. The stack typically consists of three layers: a model runtime that loads and serves the model, an orchestration layer that handles prompts, memory, and retrieval, and a user interface that lets users interact conversationally. Popular runtimes include llama.cpp, Ollama, and vLLM, while frameworks like LangChain and LlamaIndex handle orchestration. For the UI, lightweight options like Gradio or Streamlit work well for prototypes, and frameworks like Open WebUI provide a polished, ChatGPT-like experience out of the box.

Why It Matters

• Privacy: No data leaves your machine. This is critical for healthcare, legal, finance, and internal enterprise use cases.
• Cost predictability: No per-token billing. Once you have the hardware, inference is effectively free.
• Customization: You can fine-tune, quantize, and constrain the model to fit your domain.
• Offline capability: Works in air-gapped environments with no internet access.
• No vendor lock-in: Swap models freely as the open-source ecosystem evolves.

Prerequisites and Environment Setup

This tutorial assumes a machine with at least 16 GB of RAM and a modern CPU. A GPU with 8 GB or more of VRAM significantly improves performance but is not required because we will use quantized models. We will use Ollama as the runtime because it handles model downloading, quantization, and serving in one package, and it exposes an OpenAI-compatible API.

Install Ollama from the official site or via your package manager. On macOS and Linux, you can also use the install script:

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation and pull a model. We will use llama3.1 (8B parameters), which offers a strong balance of quality and resource usage:

ollama --version
ollama pull llama3.1:8b

Once downloaded, Ollama serves the model on localhost:11434 by default. You can test it with a simple curl request:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "Hello, who are you?"}],
  "stream": false
}'

Create a working directory and set up a Python virtual environment for the orchestration and UI layers:

mkdir private-chat && cd private-chat
python -m venv .venv
source .venv/bin/activate
pip install langchain langchain-community langchain-ollama \
            chromadb sentence-transformers gradio python-dotenv

Building the Core Chat Engine

The core engine wraps the Ollama model, manages conversation history, and exposes a simple interface for sending messages and receiving responses. We use LangChain's integration with Ollama and a simple in-memory list to track message history.

Creating the Chat Model Wrapper

Create a file named chat_engine.py:

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from typing import List, Optional


class ChatEngine:
    def __init__(
        self,
        model_name: str = "llama3.1:8b",
        system_prompt: Optional[str] = None,
        temperature: float = 0.7,
    ):
        self.llm = ChatOllama(
            model=model_name,
            temperature=temperature,
            base_url="http://localhost:11434",
        )
        self.system_prompt = system_prompt or (
            "You are a helpful, concise assistant. "
            "Answer accurately and ask for clarification when needed."
        )
        self.history: List = [SystemMessage(content=self.system_prompt)]

    def chat(self, user_message: str) -> str:
        self.history.append(HumanMessage(content=user_message))
        response = self.llm.invoke(self.history)
        self.history.append(AIMessage(content=response.content))
        return response.content

    def reset(self) -> None:
        self.history = [SystemMessage(content=self.system_prompt)]

    def get_history(self) -> List:
        return self.history

This class initializes a ChatOllama instance, maintains a system prompt, and appends each turn to a running history. The chat method sends the full context to the model so it can respond coherently across turns. The reset method clears history, which is useful when starting a new conversation.

Testing the Engine

Before building the UI, verify the engine works from the command line:

from chat_engine import ChatEngine

engine = ChatEngine()
print(engine.chat("What are three benefits of running models locally?"))
print(engine.chat("Can you expand on the first one?"))

Run it with python -c "import test_engine" or save it as a script. You should see context-aware responses, confirming that history is being passed correctly.

Adding Retrieval-Augmented Generation (RAG)

A bare chat model only knows what it learned during training. To let it answer questions about your private documents, we add a RAG layer. This involves embedding documents, storing them in a vector database, and retrieving relevant chunks before generating an answer.

Building the Document Store

Create rag_engine.py:

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from typing import List
import os


class RAGEngine:
    def __init__(
        self,
        model_name: str = "llama3.1:8b",
        embedding_model: str = "nomic-embed-text",
        persist_dir: str = "./chroma_db",
    ):
        os.makedirs(persist_dir, exist_ok=True)
        self.embeddings = OllamaEmbeddings(
            model=embedding_model,
            base_url="http://localhost:11434",
        )
        self.vectorstore = Chroma(
            embedding_function=self.embeddings,
            persist_directory=persist_dir,
        )
        self.llm = ChatOllama(
            model=model_name,
            temperature=0.3,
            base_url="http://localhost:11434",
        )
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=800,
            chunk_overlap=120,
        )
        self._build_chain()

    def _build_chain(self) -> None:
        prompt = ChatPromptTemplate.from_template(
            "You are a helpful assistant. Use the following context to "
            "answer the question. If the context does not contain the "
            "answer, say you don't know.\n\n"
            "Context:\n{context}\n\n"
            "Question: {question}\n\nAnswer:"
        )

        retriever = self.vectorstore.as_retriever(search_kwargs={"k": 4})

        def format_docs(docs):
            return "\n\n".join(d.page_content for d in docs)

        self.chain = (
            {
                "context": retriever | format_docs,
                "question": RunnablePassthrough(),
            }
            | prompt
            | self.llm
            | StrOutputParser()
        )

    def add_documents(self, texts: List[str], source: str = "user") -> int:
        docs = [Document(page_content=t, metadata={"source": source}) for t in texts]
        chunks = self.splitter.split_documents(docs)
        self.vectorstore.add_documents(chunks)
        return len(chunks)

    def add_file(self, path: str) -> int:
        with open(path, "r", encoding="utf-8") as f:
            text = f.read()
        return self.add_documents([text], source=path)

    def ask(self, question: str) -> str:
        return self.chain.invoke(question)

Before using this, pull the embedding model:

ollama pull nomic-embed-text

Now you can ingest documents and ask questions grounded in your own data:

from rag_engine import RAGEngine

rag = RAGEngine()
rag.add_file("company_policy.txt")
print(rag.ask("What is the remote work policy?"))

Building the User Interface

With the chat and RAG engines in place, we build a Gradio interface that provides a ChatGPT-like experience. The interface supports toggling between plain chat and document-grounded answers, uploading files, and resetting the conversation.

Create app.py:

import gradio as gr
from chat_engine import ChatEngine
from rag_engine import RAGEngine

chat_engine = ChatEngine()
rag_engine = RAGEngine()
use_rag = {"enabled": False}


def respond(message, history):
    if use_rag["enabled"]:
        reply = rag_engine.ask(message)
    else:
        reply = chat_engine.chat(message)
    return reply


def toggle_rag(value):
    use_rag["enabled"] = value
    return f"RAG mode: {'ON' if value else 'OFF'}"


def upload_file(file):
    if file is None:
        return "No file uploaded."
    count = rag_engine.add_file(file.name)
    return f"Indexed {count} chunks from {file.name}"


def reset_chat():
    chat_engine.reset()
    return [], ""


with gr.Blocks(title="Private ChatGPT") as demo:
    gr.Markdown("# Private ChatGPT — Local Models Only")
    with gr.Row():
        rag_toggle = gr.Checkbox(label="Use RAG (document-grounded)")
        status = gr.Textbox(value="RAG mode: OFF", interactive=False)
        reset_btn = gr.Button("Reset Chat")

    rag_toggle.change(toggle_rag, inputs=rag_toggle, outputs=status)
    reset_btn.click(reset_chat, outputs=[gr.Chatbot(), gr.Textbox()])

    gr.ChatInterface(
        fn=respond,
        type="messages",
        chatbot=gr.Chatbot(height=500),
    )

    with gr.Accordion("Upload Documents for RAG"):
        file_input = gr.File(label="Text file")
        upload_btn = gr.Button("Index Document")
        upload_status = gr.Textbox(label="Status", interactive=False)
        upload_btn.click(upload_file, inputs=file_input, outputs=upload_status)


if __name__ == "__main__":
    demo.launch(server_name="127.0.0.1", server_port=7860)

Launch the application:

python app.py

Open http://127.0.0.1:7860 in your browser. You now have a fully private chat application running on your own hardware, with optional document grounding.

Best Practices

Model Selection

Choose models based on your hardware and quality requirements. For machines with 16 GB RAM, 7B–8B parameter models like llama3.1:8b, mistral:7b, or qwen2.5:7b work well. If you have a GPU with 24 GB VRAM, consider llama3.1:70b quantized to 4-bit, or mixtral:8x7b for higher quality. Always benchmark latency and output quality on your own prompts before committing to a model.

Quantization

Quantization reduces memory usage by storing weights in lower precision. Ollama ships models pre-quantized (commonly to 4-bit using GGUF format). This typically reduces quality only marginally while cutting memory requirements by 4x or more. For production, test both the full-precision and quantized versions on representative tasks.

Prompt Engineering and System Prompts

The system prompt shapes behavior significantly. Be explicit about tone, length, refusal policies, and output format. For domain-specific applications, include instructions like "Answer only based on the provided context" or "Always cite the source document name." Iterate on the system prompt using a small evaluation set of representative questions.

Memory and Context Management

Every model has a context window limit (for llama3.1, it is 128K tokens, but practical limits are lower due to memory and latency). For long conversations, implement a sliding window or summarization strategy. A common pattern is to keep the last N messages and summarize older ones into a compact "conversation so far" block:

def trim_history(self, max_messages: int = 20) -> None:
    if len(self.history) <= max_messages + 1:
        return
    system = self.history[0]
    recent = self.history[-max_messages:]
    summary_prompt = (
        "Summarize the following conversation in 200 words, "
        "preserving key facts and decisions:\n\n"
        + "\n".join(
            f"{m.type}: {m.content}" for m in self.history[1:-max_messages]
        )
    )
    summary = self.llm.invoke([HumanMessage(content=summary_prompt)])
    self.history = [
        system,
        SystemMessage(content=f"Previous conversation summary: {summary.content}"),
        *recent,
    ]

Security Considerations

• Bind all services to 127.0.0.1 so they are not exposed on your network.
• If you must expose the UI remotely, put it behind a reverse proxy with authentication and TLS.
• Sanitize uploaded documents; do not ingest untrusted files without size and type checks.
• Log interactions for auditing, but store logs locally and encrypt them if they contain sensitive content.
• Be aware that local models can still produce harmful, inaccurate, or biased outputs. Implement output validation for high-stakes use cases.

Performance Tuning

For better throughput, adjust Ollama's num_ctx, num_gpu, and num_thread parameters. If you need to serve multiple concurrent users, consider vLLM instead of Ollama, as it supports continuous batching and PagedAttention. For single-user scenarios, Ollama's simplicity is hard to beat.

Persistence

The in-memory history in ChatEngine is lost on restart. For production, persist conversations to SQLite or a document database. The Chroma vector store in RAGEngine already persists to disk, so indexed documents survive restarts.

Conclusion

Building a private ChatGPT alternative with local models is now practical thanks to mature open-source tooling. With Ollama handling model serving, LangChain managing orchestration, Chroma providing retrieval, and Gradio delivering a clean interface, you can assemble a complete, privacy-preserving chat application in a few hundred lines of code. The real power emerges when you combine these components with your own documents through RAG, turning a general-purpose model into a domain-specific assistant that never sends a byte of data to the cloud. Start with the stack in this tutorial, benchmark it against your use case, and iterate on model choice, prompt design, and retrieval quality until it meets your needs.