Quantization is the single technique that made running capable LLMs on consumer hardware practical at all — it’s the difference between a model needing an $8,000 GPU and running comfortably on a gaming laptop, often with barely noticeable quality loss.
The Core Idea
A model’s weights are normally stored as 16-bit floating point numbers. Quantization reduces the precision used to store each weight — down to 8-bit, 4-bit, or even lower — shrinking the model’s memory footprint proportionally, since each weight simply takes fewer bits to represent.
A 7B parameter model, by precision: FP16 (16-bit): ~14 GB Q8_0 (8-bit): ~7.5 GB Q4_K_M (4-bit): ~4.4 GB <- the most common "sweet spot" for local use Q2_K (2-bit): ~2.6 GB <- noticeably degraded quality
Why It Doesn’t Wreck Quality (Usually)
Neural network weights are surprisingly tolerant of reduced precision — most of the “important” information lives in the relative pattern of values, not their exact numerical precision. Modern quantization methods (like GGUF’s K-quants) also apply variable precision, keeping more bits for weights that matter more and cutting harder on ones that matter less, rather than applying uniform precision loss everywhere.
Quantizing a Model Yourself
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp && cmake -B build && cmake --build build # Convert a Hugging Face model to GGUF format first python convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf # Quantize down to 4-bit ./build/bin/llama-quantize model-f16.gguf model-q4.gguf Q4_K_M
Where the Quality Loss Actually Shows Up
At 4-bit, most everyday use cases (chat, summarization, basic coding help) show minimal perceptible degradation. Precision loss becomes more noticeable on tasks demanding exact numerical reasoning or very long, complex multi-step logic — worth testing your specific use case at a couple of quantization levels rather than assuming Q4 is always “close enough.”