distillation
Distillation is a machine learning technique that trains a student model using the outputs of a stronger teacher model as its learning signal.
Distillation, also called knowledge distillation, is a machine learning technique in which a smaller or new student model learns to reproduce the behavior of a larger, more capable teacher model by training on the teacher's outputs or probability distributions. The approach was popularized by a 2015 paper by Geoffrey Hinton and colleagues.
With large language models, the term also refers to training a model on answers or reasoning traces generated by a more advanced model. It is used to build smaller, faster models and to reduce training costs.
This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.