Vector databases went from a niche research tool to core infrastructure practically overnight once RAG became mainstream. Understanding what they’re actually doing under the hood — beyond “it stores embeddings” — makes it much easier to reason about their trade-offs.
What an Embedding Actually Is
An embedding model converts text (or images, audio) into a fixed-length vector of numbers, positioned in a high-dimensional space such that semantically similar inputs end up close together. “Dog” and “puppy” land near each other; “dog” and “spreadsheet” land far apart — despite none of that similarity being explicitly programmed in.
import numpy as np
from openai import OpenAI
client = OpenAI()
def embed(text):
return np.array(client.embeddings.create(
model="text-embedding-3-small", input=text
).data[0].embedding)
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
dog = embed("dog")
puppy = embed("puppy")
spreadsheet = embed("spreadsheet")
print(cosine_similarity(dog, puppy)) # high, e.g. ~0.85
print(cosine_similarity(dog, spreadsheet)) # low, e.g. ~0.15
Why You Need a Special Index, Not Just a Loop
Finding the nearest vectors by comparing a query against every single stored vector (brute force) is O(n) — fine for a thousand documents, unusable for tens of millions. Vector databases use Approximate Nearest Neighbor (ANN) algorithms like HNSW (Hierarchical Navigable Small World graphs) to find “close enough” matches in roughly logarithmic time, trading a tiny amount of accuracy for massive speed gains.
Popular Options and Where They Fit
- pgvector — adds vector search directly to PostgreSQL; the pragmatic choice if you already run Postgres and don’t want a separate system.
- Chroma — lightweight, embeddable, great for prototyping and small-to-medium local projects.
- Pinecone / Qdrant / Weaviate — purpose-built, managed or self-hosted vector databases designed for large-scale production RAG workloads.
Hybrid Search: The Practical Sweet Spot
Pure vector search misses exact keyword matches (like specific product codes or names) that traditional full-text search handles trivially. Most production RAG systems combine both — vector similarity for semantic matching, keyword search for exact terms — and merge the results, since neither approach alone is sufficient.