Glossary · Math & training
Knowledge Distillation
Training a student model to reproduce selected behavior or output distributions from a more capable teacher, often alongside ordinary target labels.
Why it matters
It can transfer useful behavior into a smaller or cheaper model when serving the teacher directly is impractical.
In practice
Define teacher outputs, temperature, student loss, and a held-out eval set, then compare the student with both the teacher and a label-only baseline.
Common confusion
Distillation transfers behavior on the training distribution; it does not copy every capability, fact, or safety property of the teacher.
Related terms
Sources
Browse the learning paths to see this term in context — every lesson is free to read.