Model quantization formats compress a model’s weights to reduce memory footprint and speed up inference — GGUF, AWQ, and GPTQ represent genuinely different technical approaches with real, practical trade-offs worth understanding before choosing one.
GGUF: Optimized for CPU and Consumer Hardware
GGUF (used by llama.cpp and Ollama) is genuinely designed for flexible, efficient inference across a wide range of hardware, including CPU-only setups — it supports multiple quantization levels (from very aggressive 2-bit up through less aggressive 8-bit) within one broadly compatible file format, making it the natural, common choice for local, personal-machine inference.
# GGUF quantization naming shows the actual trade-off directly in the filename # Q4_K_M: 4-bit quantization, "K" quantization method, "M" (medium) variant # Lower bit count = smaller file size + faster inference, # at the real cost of somewhat reduced output quality ls models/ # llama-3.1-8b.Q4_K_M.gguf (~4.9GB) # llama-3.1-8b.Q8_0.gguf (~8.5GB) - noticeably higher quality, meaningfully larger
AWQ and GPTQ: Optimized Specifically for GPU Inference
AWQ (Activation-aware Weight Quantization) and GPTQ both target genuinely efficient GPU inference specifically, using more sophisticated calibration techniques (analyzing which specific weights matter most for actual output quality, and quantizing those particular weights somewhat less aggressively than less-important ones) to meaningfully preserve quality better than naive, uniform quantization at an equivalent overall bit rate.
Which to Choose
GGUF for local, CPU-inclusive, or genuinely mixed-hardware setups (via llama.cpp/Ollama); AWQ or GPTQ specifically for dedicated GPU-only serving where the calibration-based quality preservation genuinely matters more for your particular use case — check what your specific inference engine of choice actually supports before committing to any one particular format.