Building a Local RAG Chatbot Over Your Own Documents

Combining a local LLM with a local vector store gets you a fully offline, private chatbot that answers questions about your own documents — no API keys, no data leaving your machine, and no ongoing cost per query.

The Stack

• Ollama — serves the local LLM and a local embedding model
• ChromaDB — lightweight, embeddable local vector store
• Plain Python — no heavy framework required for a basic version

Indexing Your Documents

import ollama
import chromadb
from pathlib import Path

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("my_docs")

def embed(text):
    return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]

def chunk_text(text, size=500):
    words = text.split()
    return [" ".join(words[i:i+size]) for i in range(0, len(words), size)]

for i, path in enumerate(Path("./documents").glob("*.txt")):
    text = path.read_text()
    for j, chunk in enumerate(chunk_text(text)):
        collection.add(
            ids=[f"{i}-{j}"],
            embeddings=[embed(chunk)],
            documents=[chunk],
            metadatas=[{"source": path.name}],
        )

print(f"Indexed {collection.count()} chunks")

Querying with Local Generation

def ask(question):
    query_embedding = embed(question)
    results = collection.query(query_embeddings=[query_embedding], n_results=3)
    context = "\n\n".join(results["documents"][0])
    sources = set(m["source"] for m in results["metadatas"][0])

    response = ollama.generate(
        model="llama3.1:8b",
        prompt=f"Context:\n{context}\n\nQuestion: {question}\n"
               f"Answer using only the context above. If it's not in the context, say so."
    )
    print(response["response"])
    print(f"\nSources: {', '.join(sources)}")

ask("What does our onboarding process cover?")

Realistic Expectations

A local 8B model gives noticeably lower answer quality than a frontier hosted model, but for private, well-structured internal documents (policies, runbooks, personal notes) it’s frequently more than good enough, and the complete privacy and zero marginal cost per query are hard to beat for that use case.