znazz75.de

What Is Retrieval-Augmented Generation (RAG) and Why It Matters

Claude
,

An LLM’s knowledge is frozen at training time and it has no idea what’s in your company’s internal wiki. RAG solves both problems the same way: instead of relying purely on what the model memorized, you retrieve relevant documents at query time and hand them to the model as context.

The Basic Pipeline

  1. Split your documents into chunks and convert each into a vector embedding
  2. Store those embeddings in a vector database
  3. At query time, embed the user’s question and retrieve the most similar chunks
  4. Pass the retrieved chunks plus the question to the LLM as context, and ask it to answer using only that context

A Minimal RAG Implementation

from openai import OpenAI
import chromadb

client = OpenAI()
chroma = chromadb.Client()
collection = chroma.create_collection("docs")

def embed(text):
    return client.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

# Indexing: run once when documents change
docs = ["Our refund policy allows returns within 30 days.",
        "Support hours are 9am-5pm EST, Monday through Friday."]
for i, doc in enumerate(docs):
    collection.add(ids=[str(i)], embeddings=[embed(doc)], documents=[doc])

# Querying: run per user question
def answer(question):
    results = collection.query(query_embeddings=[embed(question)], n_results=2)
    context = "\n".join(results["documents"][0])

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {question}\nAnswer using only the context above."
        }]
    )
    return response.choices[0].message.content

print(answer("What's your refund policy?"))

Why This Beats Just Pasting Everything into the Prompt

Even models with huge context windows get measurably worse at using information buried in the middle of a very long context (“lost in the middle”). Retrieval keeps the context focused on what’s actually relevant to the specific question, which both improves accuracy and keeps costs down since you’re not paying to process irrelevant tokens on every single query.

Where RAG Falls Short

RAG is only as good as retrieval — if the relevant chunk never gets retrieved (a bad embedding match, poor chunking), the model will confidently answer without it, sometimes hallucinating. Chunk size, overlap, and retrieval quality tuning matter far more to real-world RAG quality than most people expect going in.

, ,

Recommended

Categories