Knowledge Distillation

Knowledge distillation is a training method in which a student model learns from signals produced by a teacher model. The teacher may be larger, an ensemble, a previous generation of the same architecture, or a peer trained at the same time. The transferred signal can be a probability distribution, an intermediate representation, a relation among examples, or a decoded sequence. Compression is a common goal, but it is not part of the definition: some distillation experiments use teacher and student models of equal size, and a smaller student is not automatically faster on a particular device.

The method is best understood as a family of supervised objectives rather than a procedure that literally copies a model's internal knowledge. It changes the targets or constraints used to train the student. What the student retains depends on the teacher, the transfer data, the student architecture, the loss, and the optimization process.

Historical development

An important precursor was model compression. In 2006, Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil trained a compact neural network to approximate the function learned by a large ensemble. Their method labeled a large set of real or synthetic inputs with the ensemble and trained the compact model on that pseudo-data. The paper emphasized a condition that remains central: the transfer inputs must cover the part of the input distribution on which the student will be used.^[1]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean introduced the widely used temperature-based formulation in 2015. Instead of training only on one-hot labels, their student matched the teacher's softened class probabilities. Raising the softmax temperature exposed relative probabilities among non-target classes that would otherwise be extremely small; the authors described this structure as information about how the teacher generalizes.^[2]

Subsequent work broadened both the transferred signal and the teacher–student arrangement. A peer-reviewed survey groups methods by response, feature, and relation knowledge and by offline, online, and self-distillation schemes. These labels are useful for organization, but they overlap: one training recipe can combine several signals and more than one scheme.^[3]

Core objective

Soft targets and temperature

For a classification input $x$, let $z_i^T(x)$ and $z_i^S(x)$ be the teacher and student logits for class $i$. At temperature $\tau > 0$, their softened distributions are

$$\begin{aligned} p_\tau(i\mid x) &=\frac{\exp(z_i^T(x)/\tau)} {\sum_j \exp(z_j^T(x)/\tau)},\\ q_\tau(i\mid x) &=\frac{\exp(z_i^S(x)/\tau)} {\sum_j \exp(z_j^S(x)/\tau)}. \end{aligned}$$

When $\tau=1$, these are ordinary softmax distributions. Increasing $\tau$ makes each distribution less concentrated. In the original formulation, the teacher produces targets at a chosen high temperature, the student matches them at the same temperature, and the trained student returns to temperature 1 for prediction.^[2]

A common loss combines a teacher-matching term with ordinary supervised learning:

$$\mathcal{L} = \alpha\,\tau^2 H(p_\tau,q_\tau) + (1-\alpha)\,H(y,q_1),$$

where $H$ is cross-entropy, $y$ is the ground-truth target, and $\alpha$ controls the mixture. With a fixed teacher, minimizing $H(p_\tau,q_\tau)$ is equivalent, up to a constant independent of the student, to minimizing the forward Kullback–Leibler divergence $D_{\mathrm{KL}}(p_\tau\|q_\tau)$. Hinton and colleagues multiplied the soft-target term by $\tau^2$ because its gradient magnitude scales approximately as $1/\tau^2$ in their combined objective. The factor is not a universal convention for every later distillation loss.^[2]

The hard-label term and teacher term answer different questions. The first asks the student to predict the observed label. The second asks it to behave like the teacher over the classes. Their weights therefore control a real trade-off: a student can improve task accuracy without closely reproducing the teacher, or can become more faithful to the teacher without improving on held-out data.

What the loss can and cannot transfer

The mathematical target is an observable behavior or representation, not an explanation of how the teacher reached it. A response loss can align output distributions while leaving the internal computations very different. Conversely, an intermediate-feature loss can constrain selected layers but does not reproduce all teacher parameters or computations. Distillation should therefore not be described as transferring weights, memories, or reasoning processes unless a method actually exposes and constrains those objects.

By construction, the teacher-matching term also does not distinguish an informative teacher preference from a systematic teacher error. Ground-truth supervision, filtering, other objectives, or a different teacher can counterbalance such errors; the basic loss cannot identify them on its own.

Transfer signals

Scroll sideways for more →

Feature, attention, and relation matching

FitNets extended output matching by using a teacher's intermediate representation as a hint. Because the teacher and student layers can have different dimensions, the method learns a regressor that maps the student's guided layer to the teacher's hint layer before applying a squared-error loss. The original procedure first trained the guided portion of the student with the hint and then trained the full model with output distillation.^[4]

Attention transfer defined activation-based attention maps for convolutional neural networks and trained the student to match the teacher's maps. In that work, “attention” is a specific summary computed from convolutional activations; it should not be conflated automatically with every mechanism named attention in later architectures.^[5]

Relational knowledge distillation instead matches structure among examples. Park and colleagues proposed distance-wise and angle-wise losses over teacher and student representations. The target is therefore not one activation vector per example but a pattern of pairwise or higher-order relations within a batch.^[6]

Feature-based methods introduce design choices that response matching can avoid: which layers correspond, how dimensions are projected, whether spatial resolutions agree, and how multiple losses are scaled. They can provide richer constraints, but only when the relevant teacher internals are available.

Teacher–student arrangements

Offline distillation

In the conventional offline setup, the teacher is trained first and held fixed while the student is optimized. Teacher outputs may be computed during training or cached in advance. Caching saves repeated teacher evaluation but fixes the teacher target for each stored input and augmentation. Running the teacher online costs more computation but permits the same transformed input to be shown to both models.

Online and mutual learning

A fixed, pre-trained teacher is not mandatory. Deep Mutual Learning trains a group of initially untrained networks together; each network combines its supervised loss with a mimicry loss based on the predictions of its peers. In the authors' experiments, this peer-teaching arrangement improved the participating networks without a separate high-capacity teacher.^[7] This is commonly classified as online distillation, though the peers need not form a smaller-student/larger-teacher hierarchy.

Self-distillation

In self-distillation, supervisory signals come from the same architecture, an earlier generation, or a deeper part of the same network. Born-Again Networks trained a new model with the same architecture as a converged predecessor and reported improved validation performance in their tested vision and language-modeling settings.^[8] Because teacher and student had equal capacity, this result is evidence that distillation can act as a training or regularization method; it is not model compression by itself.

Distillation for sequence and language models

Token and sequence targets

For an autoregressive model, a teacher can supervise the distribution over the next token at each position. This is the closest analogue of class-probability distillation. It requires compatible output spaces or an explicit alignment when teacher and student use different tokenizations.

Sequence-level knowledge distillation uses a different target. Kim and Rush decoded translations from a teacher with beam search and trained the student on selected teacher sequences. Their method converts a distribution over many possible sequences into one or a small number of pseudo-targets.^[9] This can be used when a full sequence is available but the teacher's complete token distribution is not. It also means that decoding settings and filtering become part of the distillation procedure.

Distilled encoder models

Several compact Transformer encoders illustrate how multiple objectives and architectural changes can be combined:

Scroll sideways for more →

These are results from the papers' particular tasks, software, and hardware, not general compression ratios. DistilBERT applies distillation during pre-training rather than only to a task-specific classifier.^[10] TinyBERT combines output, hidden-state, and attention objectives at both general and task-specific stages.^[11] MobileBERT's result depends on architectural engineering and a purpose-built teacher as well as knowledge transfer.^[12]

Large language models and output-only distillation

Large generative models have widened the use of output-only, sometimes called black-box, distillation. If only generated text is available, the student can learn from responses or rationales, but it cannot match unavailable teacher logits or hidden states. MiniLLM explicitly distinguishes that setting from white-box distillation. Its own white-box method optimizes reverse $D_{\mathrm{KL}}(q\|p)$ between student and teacher sequence distributions, rather than the forward divergence used by many standard objectives, and samples from the student during training.^[13]

Other methods treat teacher-generated reasoning traces as auxiliary supervision. Distilling Step-by-Step trains a smaller model jointly on task labels and rationales produced by a larger model. Its reported gains are tied to four evaluated natural-language benchmarks and should not be read as a general result that generated rationales always improve students.^[14]

The DeepSeek-R1-Distill models provide a useful naming boundary. The DeepSeek-R1 paper says that the corresponding Qwen and Llama base models were fine-tuned for two to three epochs on an 800,000-example supervised dataset: about 600,000 reasoning-related samples and about 200,000 non-reasoning samples in the described pipeline.^[15] The paper calls the resulting models “distilled,” but the documented procedure is supervised fine-tuning on curated outputs. That evidence does not establish transfer of DeepSeek-R1's weights, hidden states, or complete token distributions.

For output-only methods, the teacher query set, decoding parameters, selection rules, rejected samples, and final training mixture are part of the method. A model name alone does not show which of those controls were used.

Distillation for diffusion models

Distillation can target the inference procedure rather than only model size. Progressive distillation starts from a deterministic diffusion model sampler and trains a new model to reproduce two sampling steps in one. Repeating the procedure halves the step count at each stage. Salimans and Ho reported models using as few as four sampling steps in their image-generation experiments while maintaining much of the measured sample quality.^[16]

Consistency models can also be trained by distilling a pre-trained diffusion model so that points along a probability-flow trajectory map to the same clean-data endpoint. This supports one-step or few-step generation. However, the same paper also trains consistency models without a pre-trained teacher.^[17] “Consistency model” and “distilled diffusion model” are therefore not synonyms.

These examples compress a computation path: the central deployment measure is the number and cost of sampling evaluations, not merely the number of parameters.

Evidence, limitations, and failure modes

Results are conditional

There is no source-supported universal ratio for how much distillation reduces model size, latency, or accuracy loss. The student architecture sets the parameter count; the execution stack and hardware determine latency; and benchmark results depend on the teacher, data, objective, and evaluation protocol. Reported numbers should retain their experimental context.

Capacity and architecture gaps

A more accurate or larger teacher is not necessarily a better teacher for a particular student. Cho and Hariharan found that students in their image-classification experiments could have difficulty matching much larger teachers; teacher accuracy was a poor predictor of student performance, and early stopping the teacher sometimes helped.^[18] A later Teacher Assistant study reported that inserting an intermediate-capacity model improved results in its tested CNN and ResNet settings.^[19] These findings are not a single guaranteed recipe: they show that the capacity path is an empirical design choice.

Feature transfer adds another mismatch. Layers can differ in width, spatial resolution, depth, or semantic role. A learned projector resolves dimensional compatibility, not semantic equivalence. Layer mappings therefore need ablation rather than being assumed correct.

Fidelity is not generalization

Stanton and colleagues separated fidelity, meaning agreement with the teacher's predictive distribution, from generalization on unseen data. They found substantial teacher–student discrepancies even when a student had enough representational capacity, identified optimization as one cause, and observed that closer teacher matching did not always yield better generalization.^[20] Distillation can help a student without making it a behavioral replica of the teacher.

This distinction affects evaluation. Top-line task accuracy alone cannot show whether the student preserved teacher behavior. Conversely, a low divergence on transfer data cannot show that the student generalizes well outside that data.

Transfer-data coverage

The student only observes teacher behavior on the transfer inputs. Buciluă and colleagues already noted that pseudo-data outside the relevant input manifold wastes samples, while poor coverage misses important regions.^[1] Stanton and colleagues likewise found that the distillation dataset affects teacher–student agreement.^[20] Unlabeled data is useful only to the extent that it represents the deployment distribution and can be labeled reliably by the teacher.

Input consistency and cached targets

Data augmentation can silently change the target being matched. In a computer-vision study, Beyer and colleagues found that showing teacher and student the same randomly transformed view was important in their setup; precomputed targets for a fixed view could generalize poorly when the student saw a different crop.^[21] The broader lesson is to define whether the target belongs to the original input, the transformed input, or an average over transformations.

Training and access costs

Distillation usually lowers the student's inference cost only after adding a training stage. Offline methods may require training and storing a teacher, evaluating it over a transfer set, and optimizing one or more students. White-box methods additionally require access to logits or internal states; output-only methods may require many generations and filtering. These costs can be worthwhile for repeated deployment, but they should be reported separately from student inference cost.

Teacher dependence

Because the teacher output is a training target, systematic teacher errors can be reinforced unless labels, filtering, or other constraints oppose them. The student's smaller capacity may remove some behavior rather than preserve it. Claims that distillation transfers safety, calibration, robustness, or domain knowledge require separate measurements for those properties; task accuracy is not a substitute.

Relationship to other compression methods

Scroll sideways for more →

Pruning and quantization operate directly on a model's parameters or numeric representation. Han, Mao, and Dally's Deep Compression pipeline, for example, combined pruning, trained quantization, and Huffman coding.^[22] Distillation instead trains a predictive model from teacher-provided targets. The methods are complementary: a student can be distilled, then pruned or quantized, or distillation can be used while recovering performance after those transformations.

Parameter count, file size, peak memory, throughput, single-request latency, and energy are different measurements. A valid comparison states which one was optimized and measures it on the intended runtime.

Experimental practice

A reproducible distillation study should specify:

1. Deployment objective. Record the target model size, latency, throughput, memory, energy, or sampling-step budget and the hardware and software used to measure it.
2. Teacher and student identities. Preserve exact checkpoints, architectures, parameter counts, tokenizers, output spaces, and initialization procedures.
3. Transfer data. Document the source, sampling, augmentation, filtering, labels, and relationship to the final evaluation distribution.
4. Teacher access. State whether training used decoded outputs, top-$k$ probabilities, full logits, hidden states, attention maps, or another representation.
5. Objective. Report temperature, divergence direction, loss weights, layer mappings, projectors, normalization, and schedules. For generated targets, include decoding and selection settings.
6. Ablations. Compare label-only training, teacher-only training, the combined objective, and the relevant architectural or data controls.
7. Evaluation. Measure the student's task performance and deployment cost. When behavior matching matters, also measure teacher–student fidelity on held-out and shifted inputs.
8. Accounting. Separate teacher training, teacher inference or API queries, student training, and student inference costs.

The strongest conclusion supported by a successful experiment is specific: a named student, trained from a named teacher with a documented signal and dataset, met measured quality and deployment targets. Broader claims require replication across teachers, students, tasks, and runtimes.

See also

• Model compression
• Ensemble learning
• Transfer learning
• Deep learning
• Large language model
• Natural language processing
• Computer vision

References

1. ^Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. “Model Compression.” Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541, 2006. doi:10.1145/1150402.1150464.
2. ^Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. “Distilling the Knowledge in a Neural Network.” arXiv:1503.02531, 2015.
3. ^Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. “Knowledge Distillation: A Survey.” International Journal of Computer Vision 129, pp. 1789–1819, 2021.
4. ^Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. “FitNets: Hints for Thin Deep Nets.” International Conference on Learning Representations, 2015.
5. ^Sergey Zagoruyko and Nikos Komodakis. “Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer.” International Conference on Learning Representations, 2017.
6. ^Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. “Relational Knowledge Distillation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967–3976, 2019.
7. ^Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. “Deep Mutual Learning.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320–4328, 2018.
8. ^Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. “Born Again Neural Networks.” Proceedings of the 35th International Conference on Machine Learning, PMLR 80, pp. 1607–1616, 2018.
9. ^Yoon Kim and Alexander M. Rush. “Sequence-Level Knowledge Distillation.” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, 2016.
10. ^Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. “DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter.” 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019.
11. ^Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. “TinyBERT: Distilling BERT for Natural Language Understanding.” Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163–4174, 2020.
12. ^Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. “MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158–2170, 2020.
13. ^Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. “MiniLLM: Knowledge Distillation of Large Language Models.” International Conference on Learning Representations, 2024.
14. ^Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. “Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes.” Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003–8017, 2023.
15. ^DeepSeek-AI et al. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv:2501.12948, revised January 2026.
16. ^Tim Salimans and Jonathan Ho. “Progressive Distillation for Fast Sampling of Diffusion Models.” International Conference on Learning Representations, 2022.
17. ^Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. “Consistency Models.” Proceedings of the 40th International Conference on Machine Learning, PMLR 202, pp. 32211–32252, 2023.
18. ^Jang Hyun Cho and Bharath Hariharan. “On the Efficacy of Knowledge Distillation.” Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4794–4802, 2019.
19. ^Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. “Improved Knowledge Distillation via Teacher Assistant.” Proceedings of the AAAI Conference on Artificial Intelligence 34(04), pp. 5191–5198, 2020.
20. ^Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. “Does Knowledge Distillation Really Work?.” Advances in Neural Information Processing Systems 34, 2021.
21. ^Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. “Knowledge Distillation: A Good Teacher Is Patient and Consistent.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10925–10934, 2022.
22. ^Song Han, Huizi Mao, and William J. Dally. “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding.” International Conference on Learning Representations, 2016.