How Big Models Teach Small Models to Be Smart

Knowledge distillation lets a small AI model learn from a larger, more expensive one by copying its behavior rather than compressing it. The larger "teacher" model produces detailed probability outputs, called soft labels, that reveal relationships between possible answers, giving the smaller "student" model richer training signals than plain data alone. This approach now powers real-world models like Google's Gemma, and in 2025, DeepSeek used it to build a 7-billion-parameter model that outperformed a 32-billion-parameter model on math benchmarks.