Model distillation trains a smaller “student” model to mimic a larger “teacher” model’s behavior — a genuinely practical technique for getting most of a large model’s capability into something small enough to actually deploy cheaply.
1. Train (or use an existing) large, high-capability "teacher" model 2. Have the teacher generate outputs (or full probability distributions, not just the final answer) on a large set of inputs 3. Train a much smaller "student" model to match the TEACHER's outputs, not the original raw training labels 4. The student ends up capturing much of the teacher's learned behavior, at a fraction of the parameter count and inference cost
Why Matching the Teacher Beats Training from Scratch
Training a small model directly on raw labels alone often underperforms a distilled model of the same size — the teacher’s full probability distribution over possible answers (not just its single top answer) carries genuinely richer information (“this could also plausibly be X, less likely Y”) that helps the smaller student learn more efficiently than raw labels alone would allow.
Where This Shows Up in Practice
Many of the smaller, faster models offered alongside a flagship model (a “mini” or “lite” variant) are frequently produced via distillation from the larger flagship — genuinely capturing a meaningful fraction of the larger model’s capability at dramatically lower inference cost and latency, which is exactly the trade-off that makes running AI on constrained hardware or at high request volume practical.