Distillation Can Make AI Models Smaller and Cheaper

Distillation, a decade-old AI technique pioneered by Geoffrey Hinton and Google researchers, allows smaller "student" models to learn from larger "teacher" models by capturing probability distributions rather than just right/wrong answers. This process has become fundamental in creating more efficient AI systems, enabling companies like DeepSeek to develop powerful chatbots that rival industry giants while using significantly less computing power and cost.