Knowledge Distillation: Making a Small Student Approach the Teacher's Conditional Distribution and Behavior

The core problem knowledge distillation solves is: when transferring the judgment ability of a larger or stronger model to a smaller model, what information exactly should be transferred. The simplest transfer method uses only hard labels—for each input the teacher gives only the final correct class, for example a refund case label written refund=1, ask=0, reject=0. This one-hot encoding conveys only "the answer is refund"; the information for all other classes becomes zero.

Look at a concrete comparison. Two teachers process the same refund case, and both give the final answer refund, but teacher A gives the ask class a probability of 35%, while teacher B gives only 1%. If we look only at hard labels, the labels of the two are exactly the same; the signal the student learns from them has no difference. However, the facts conveyed by these two judgments are completely different: giving ask 35% means the case is near the decision boundary of "insufficient information, should first ask the user", and the evidence has an area of ambiguity; giving only 1% means the evidence clearly points to a refund. What the soft distribution preserves is precisely this relative relationship between classes and boundary information, while hard labels compress them into a single spike.

The mechanism of knowledge distillation is to give the student the teacher's complete probability distribution rather than a single answer. The input is still the same refund case; the teacher outputs the probabilities of the three classes refund, ask, and reject, forming a probability table, and the student model fits to this table as the supervised target. The causal chain is: the teacher produces a complete soft distribution for each input → the soft distribution carries class similarity and decision boundary information → the student fits the entire distribution rather than isolated point labels → the student gets a smoother, more sample-efficient supervisory signal than one-hot labels, because each sample conveys gradient information about all classes, not just the correct class.

The closer the student output is to the teacher distribution, it only shows that behavior transfer has succeeded; it absolutely does not show that the teacher's judgment is the ground truth. The teacher's own probabilities may be miscalibrated; "darker knowledge" does not automatically equal truth. Therefore hard ground truth and teacher signal should be used in combination, and calibrated on real labels, so that the teacher soft distribution supplements rather than replaces hard labels.

The object of distillation is not limited to probability distributions either: it can fit logits, hidden representations, attention, sequence demonstrations, preferences, or verification feedback. Each kind of observable information conveys different content, and which one to choose depends on which part of the teacher's behavior the student needs to inherit.

The reason the teacher's soft distribution is more informative than hard labels is not mainly how high the probability is for the class the teacher got right, but that when it is wrong, differences among the other classes are still preserved. At T=1, softmax crushes the largest logit to a probability near 1 and the remaining classes near 0, so the student sees almost only "answers" rather than "which mistakes are closer to the correct answer". The introduction of temperature T solves this problem: the temperature-scaled probability for class i, pᵀᵢ, is computed by applying softmax to zᵢ/T, where zᵢ is the unnormalized raw score (logit) for class i, i is the class index, and T is the temperature; softmax converts z/T into a probability distribution that sums to 1. When T>1, all logits are shrunk simultaneously, the score differences between classes narrow, and the distribution becomes flatter: the suboptimal classes that were previously crushed near zero are "pulled out" and receive nonzero probabilities. The student can then read the teacher's relative preferences—a slightly higher probability on a wrong class indicates that the teacher considers it "comparatively similar" to the correct answer; these low-probability classes also produce learnable gradients, rather than sitting at near-zero probability positions with no signal. After training, at inference time it is common to revert to T=1 so that the student outputs final probabilities on the original logit scale.

The gradient scale issue introduced by temperature must be handled in the loss function. The student's total training loss L consists of two weighted parts: L = α × L_hard_label + (1−α) × T² × KL(p_teacher ∥ p_student). L_hard_label measures the gap between the student output and human-provided ground truth; α ranges from 0 to 1 and controls the ratio of ground-truth supervision to teacher supervision; the larger α is, the more the student relies on hard labels, and the smaller α is, the more it relies on the teacher's soft distribution. KL(p_teacher ∥ p_student) measures how much the student's soft distribution deviates from the teacher's soft distribution. The derivative of softmax shrinks as temperature rises, causing the gradient scale to become smaller; multiplying by T² compensates for this effect. Note that KL divergence is not a symmetric distance: p_teacher ∥ p_student and p_student ∥ p_teacher are different, and swapping the two sides changes the emphasis of the penalty. The classical direction penalizes classes where "the teacher gives nonzero probability but the student misses them" more strongly, which encourages the student to cover the teacher's entire distribution; reverse KL more easily makes the student concentrate on a few high-probability modes. Therefore, when reporting a distillation scheme, you cannot just say "used KL"; you must specify the direction and the sample unit. For generative models, you can also switch to sequence-level objectives, and then the choice of direction and unit is even less dispensable.

The boundary of temperature's role must also be made clear: it is a training-signal softening mechanism, not a calibration fix. It does not guarantee that the teacher's or student's probability outputs are trustworthy at deployment; whether they are calibrated needs to be handled separately.

Use a customer service scenario to ground the effect of temperature in concrete numbers. The teacher faces three refund actions—refund, ask, and reject—and outputs raw logits of 4, 2, and 0, respectively. With T=1, applying softmax directly gives probabilities e⁴/Σ ≈ .867, e²/Σ ≈ .117, and e⁰/Σ ≈ .016, where Σ = e⁴ + e² + e⁰. The teacher puts almost all probability on refund, while ask and reject receive only about one tenth and less than two percent respectively; the student can hardly read the "second-best choice" information from this distribution. When the temperature is raised to T=2, each logit is first divided by 2 before entering softmax, so the probabilities become e²/Σ' ≈ .665, e¹/Σ' ≈ .245, and e⁰/Σ' ≈ .090, where Σ' = e² + e¹ + e⁰. In the bar chart corresponding to Figure 1, you can see that temperature 2 does not change the ranking of the three actions: refund remains highest, ask second, reject lowest, but ask's probability rises from .117 to .245, reject's from .016 to .090, and the gap between each and refund clearly narrows. After temperature softening, the relative information of ask/reject becomes visible: reject has a lower probability than ask, indicating that in the teacher's view reject is less close to the correct answer than ask—a relationship that is almost flattened to invisibility at T=1.

What practical benefit does this softened distribution bring to the student? We can compare it with the student's output. Suppose the student outputs [.60, .30, .10], meaning it gives 30% probability to ask and 10% to reject. Compared with the teacher distribution at T=2 of .665/.245/.090, the student's allocation direction is consistent with the teacher's, only more even across the three actions, so the soft distribution loss will be quite small and the student gets a clear positive signal: preserving probability for ask and reject is encouraged. If training uses only hard labels, the only true target is refund, and the loss will keep pushing the student to drive refund to 1 and ask and reject to 0; the relative relationships the student just learned would instead be punished as errors. This is the difference between the two training signals for the same teacher output: hard labels reward only the answer itself, while soft distributions also reward "differences among wrong ways".

When the teacher's internal state is unavailable, what else can be distilled depends on which signal the teacher is willing to expose. According to the level at which the signal resides, we can distinguish four types: logit is the raw candidate score before softmax, that is, the unnormalized score for each class or vocabulary entry; feature is the vector produced by the teacher's intermediate layers; relation is the similarity structure between samples, or between layers; sequence is a complete answer ultimately generated by the teacher. The more internal the signal, the finer the information it provides, but the more it requires access to the inside of the teacher model; there is a clear trade-off between the two.

Looking at the four levels one by one: The Logit/token level obtains the distribution over each class or vocabulary, and the student can directly perform regression or KL matching on this distribution, provided that the teacher's logits are accessible and the vocabularies of both are aligned; however, public APIs often do not expose full logits, so this premise often fails. The Feature level obtains vectors from hidden layers or attention, requiring layer mapping first and then dimension projection before the student can match; when architectures differ greatly, the mapping itself is difficult. The Relation level obtains similarity relationships between samples or between layers, typically transferred through in-batch contrastive learning; the objective is indirect and does not directly give each sample a "correct answer". The Sequence level obtains the teacher's answers, explanations, or preference demonstrations, requiring only a text-generation API, which is almost universally usable, but the student can observe only the single path that the teacher ultimately sampled.

The inputs for choosing a distillation method are three practical conditions: the available teacher interface, the student's architecture, and the capability one wants to retain; the outputs are the corresponding probability, feature, relation, or demonstration data, plus a matching student loss. Large language models commonly use sequence distillation: have the teacher generate demonstrations for diverse prompts, then train the student with supervised fine-tuning or preference training. This approach is essentially more like behavior cloning—the student imitates the teacher's "way of speaking" but cannot see the teacher's probability distribution over unsampled tokens, so it cannot know why the teacher did not choose another word; therefore it easily copies the teacher's style and verbosity as well. An observable symptom is: the student performs well on inputs highly consistent with the demonstration style, but fails as soon as the wording changes. At this point, it should be explained that the student only copied the sampled path, rather than fully transferring the teacher's distribution. Architectural differences bring another limitation: different models may not have layers that can be mapped one-to-one directly; forcing layer-by-layer feature matching at this point would actually restrict the student from developing its own representation style, turning distillation into a copy of the teacher's structure.

Why can't all of the teacher's capabilities be packed into the student simply by "demonstrating enough"? Because the student's own parameter scale, number of layers, context window, vocabulary, and computational depth directly limit the representations and algorithms it can express. Distillation is not lossless transfer; it is a biased fit of teacher behavior under the student's own capacity constraints. The training objective naturally forms a retention priority: distilled data makes the student preferentially retain behaviors that occur frequently, are easy to fit, and have high weight in the objective function; correspondingly, rare reasoning patterns, long-context capabilities, calibration quality, adversarial robustness, and tool recovery abilities are often the first to be discarded—either because they are scarce in the data, or because they are costly to fit and contribute little to reducing the loss.

When capacity is insufficient, continuing to add teacher samples does not improve the student linearly. Gains enter a plateau: for the portion of teacher outputs that exceeds the student's expressive range, no amount of additional samples can fit it; and when behaviors from different tasks conflict with each other, stacking samples can even create interference, pulling the student back and forth between two directions. Therefore, sensible expectation management is to draw the capability–parameter–latency frontier curve, clarifying how much can be retained under a given parameter and latency budget, rather than treating "full replication of teacher capabilities" as the default promise.

The same logic also explains a seemingly contradictory phenomenon: a student surpassing the teacher on a particular benchmark does not mean it is comprehensively stronger. The excess may come from regularization effects during student training, from the student coincidentally matching the data distribution of that benchmark better, or from evaluation contamination—benchmark questions or variants of them appearing in the training data. A single number cannot support a conclusion of "comprehensive superiority"; it needs to be judged comprehensively by combining the capability frontier and the lost items.

If the strongest teacher only teaches on short English FAQs, what kind of student do you get? The answer is straightforward: the student will approximate the teacher on this short English FAQ distribution, but will not spontaneously acquire long-context, multilingual, tool-calling, refusal, or error recovery capabilities. Which behaviors the distillation data covers determines which behaviors the student learns; for behaviors not covered by the data, no matter how strong the teacher is, they cannot be transmitted. The student's unevenness is determined by data coverage, not by the teacher's brand.

Therefore, distillation data should proactively construct a curriculum according to target behaviors: introduce real traffic distributions, difficult samples, high-risk scenarios, and adversarial sets, and have the teacher output on these slices. At the same time, there is a reverse discipline: for samples where the teacher is uncertain, or samples that fail verification, keep human ground truth and do not force the student to imitate the teacher's errors. The entire data pipeline can be expanded in order: first, draw real or synthetic prompts according to target slices, covering every behavior you want to retain; then have the teacher do diversified sampling, and keep the corresponding output probabilities, model version, and prompt to ensure traceability; next, use external rules, executors, or human verification to filter out errors in the teacher's answers; then deduplicate, and control difficulty and teacher style to avoid the student learning only one tone; finally, validate on a real set that does not contain teacher nearest neighbors—if the validation samples are highly similar to the training samples, the evaluation results will overestimate transfer effects.

Teacher queries themselves have cost, so sampling should not be wasted uniformly. A more economical approach is active selection: prioritize asking samples where the student is most uncertain and the teacher may have gain. The student needs supervision signals most on these samples, and the teacher's answers have the highest marginal value; for distributions where the student already answers well, repeated teacher queries bring little information gain. Active selection focuses limited teacher budget on the boundary of the student's abilities.

Are smaller and cheaper students naturally safer? No. Distillation transmits teacher behavior, not filtered advantages. Demonstration sequences may contain hallucination, privacy leakage, bias, or excessive refusals, and the student will learn them anyway; soft distributions likewise transmit the teacher's preference structure, including undesirable preferences. Another hidden pitfall is the verifier: if the teacher and verifier share the same origin—for example, the verifier is the teacher model itself or was trained on teacher data—then a high pass rate may simply be the shared blind spot of both, and verification provides no independent information. Therefore, for sensitive tasks, the supervision signal cannot come from the teacher alone; it needs to combine hard labels, independent rules, disagreement among multiple teachers, and human review.

Insufficient capacity also creates the reverse safety problem. If the student learns only surface-level safety wording without learning the conditional judgment of when to refuse, two failure modes appear: missing refusals for dangerous requests, or blanket refusals for normal requests. The former is a failure of the safety fallback, and the latter is a usability collapse; both stem from the student losing the conditional structure behind the teacher's behavior. During evaluation, you should compare the teacher, student, and ground-truth labels across risk slices, rather than using "student-teacher agreement rate" as the safety ground truth—agreement with the teacher does not equal agreement with the correct safety boundary.

Distillation can also wash away uncertainty. If demonstrations use only a single teacher's greedy answer, the student sees only the one path the teacher ultimately selected, not the teacher's probability mass over other answers, and therefore cannot see the multimodal distribution of the problem itself. The result is that the student states things more certainly than the evidence allows, flattening the teacher's internal hesitation into a decisive tone. This illusion of certainty is especially dangerous on open-ended questions, because it disguises "the teacher's most likely statement" as the "only fact".

The four methods all say they make models "smaller," but they change completely different things, and their benefits and loss sources are also different. Distillation changes the behavioral subject: it trains a smaller student to approximate the teacher's behavior. Its main benefits are reductions in parameter count and FLOPs, and its main loss source is the gap between student capacity and data coverage. Quantization changes numerical bit width: it compresses weights or activations from high-precision representation to low precision. Its main benefits are storage and bandwidth, and its main loss source is rounding and truncation error. Pruning changes structure: it directly removes weights or entire structural units. Its main benefits are sparsity and reduced compute, and its main loss source is mistakenly deleting important connections. LoRA does not change the original model but rather the task increment: it adds low-rank adaptation matrices in a bypass. Its main benefits are smaller training state and faster switching between tasks, and its main loss source is limited adaptation capacity and interference between tasks. The common point of the four is that they all reduce deployment cost, but they compress in different dimensions and cannot replace one another.

They can be combined, for example first distilling to obtain a small model, then quantizing that small model. But combining is not equivalent to simply adding errors: quantization may happen to exacerbate ability slices where the student is already fragile—on some distributions the student originally fits only marginally, and numerical error can make these slices collapse first; conversely, if the teacher used for distillation is itself quantized, the teacher's output distribution has already changed, and the student learns a signal distorted by quantization. Therefore, any combined scheme must be re-evaluated end-to-end; one cannot take a separately validated distillation result and a separately validated quantization result and piece together a conclusion that "should also hold."

With 70% fewer student parameters, how do you prove it is not "cheaper but fails more often"? You cannot rely on a single benchmark score, but must report two layers of evidence simultaneously. The model capability layer must cover: per-token KL divergence or perplexity, the target task itself, long context, multilingual, tool calling, safety, calibration quality, and out-of-distribution slices; the system cost layer must measure TTFT/TPOT (first token and per output token latency), throughput, memory footprint, energy consumption, human takeover rate, and cost per successful task. Both layers are indispensable: capability metrics answer "how well did it learn", and system metrics answer "whether the savings are real". The baseline also must be chosen correctly—not compared with the teacher, but with strong model routing or quantization schemes under the same budget, so that what is compared is "whether I made the best choice at this cost point".

Per-sample comparison can locate the position of the loss. Divide evaluation samples by outcome into three categories: teacher-only-correct (teacher correct, student wrong), student-only-correct (student correct, teacher wrong), both-wrong (both wrong), and observe which slices the capability loss concentrates in. There is a bookkeeping discipline here: samples that the teacher itself answers incorrectly cannot be counted as the student's "failure to imitate"—if the student imitates an error it is not a distillation failure, and capabilities the teacher does not have were never within the transferable range. After release is not the endpoint either: online traffic will drift, and distributions not present at training time will appear; at that point either re-distill or upgrade the routing strategy to send difficult requests back to a stronger model.

Based on this, a criterion for "acceptable compression" can be given: all hard risk thresholds are passed; the benefits on target hardware are real and measurable, not paper parameters; the sacrificed capabilities are explicitly listed, and can either be routed to other models or rolled back. Only when all three are satisfied is the compression acceptable; missing any one of them merely shifts the cost to unmeasured places.

Connect the whole chain and you can see that knowledge distillation moves from a vague "make small models stronger" aspiration to a verifiable process with an explicit order. The first step is to define capability slices: clarify which capabilities the student must retain—target tasks, long context, multilinguality, tool calling, safety behavior, calibration, and so on—and list them item by item. Without this step, all later claims that something "works well" have no baseline for comparison. The second step is for the teacher to generate signals: the teacher outputs distributions, representations, or demonstrations for these slices, and the signal type is determined by the available interface. The third step is to independently verify and construct a coverage curriculum: use rules, executors, or human verification to filter out teacher errors, and supplement real traffic, hard examples, and adversarial examples according to the slices, ensuring that data coverage corresponds one-to-one with capability slices. The fourth step is joint student fitting: under the student's own capacity constraints, simultaneously fit the hard labels and teacher signals, using temperature and loss weighting to control the ratio between them. The fifth step is per-sample comparison: classify according to teacher-only-correct, student-only-correct, and both-wrong, locate where capabilities transfer and where they are lost, and account separately for samples where the teacher answers incorrectly. The sixth step is a joint system benefit and risk gate: capability metrics are accepted together with cost metrics such as TTFT/TPOT, throughput, memory, energy, and human takeover rate; sacrificed capabilities must be explicitly listed and either routable or recoverable through fallback, and an upgrade path must be retained to handle post-release traffic drift. The six steps form a closed loop: each subsequent step checks whether the previous step's promises are fulfilled. If any link is skipped, distillation's "cheaper" may simply shift costs to unmeasured failures.