How Distillation Works: From Teacher to Student
The five-step pipeline, soft labels, and temperature; using the six distilled models DeepSeek open-sourced alongside R1 as the sample
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTHow Distillation Works: From Teacher to Student?
The five-step pipeline, soft labels, and temperature; using the six distilled models DeepSeek open-sourced alongside R1 as the sample
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Assemble the question set
Collect questions that cover the target domain. You can curate them by hand or have a model generate them. This step sets the ceiling on what the student can do: whatever domain you never ask about, the student never learns.
The teacher answers, and you keep the full reasoning
The word that matters is "full." Keep only the final answer and the student just memorizes conclusions; keep the intermediate reasoning chain and it has a chance to learn how to think. That is also why reasoning models make particularly good teachers — they already write their thinking out.
Take the probability distribution, not just the final answer
Every time the model emits a word, what exists inside it is a full probability table over candidate words. Taking only the top entry throws away a lot of information; keeping the whole table transfers a completely different amount. The next section expands on this.
Train the student on that material
The training data has three parts: the original questions, the teacher's complete answers, and the probability distribution at every step. The student's objective is to bring its own output distribution as close as it can to the teacher's.
Evaluate and iterate
Measure how far the student is from the teacher on a test set, then go back and adjust the question set's coverage and the training configuration. This usually takes several rounds.
Say you have an image classification question where the correct answer is cat. Conventional training tells the model: cat is right, everything else is wrong. Distillation hands the teacher's judgment to the student as-is.
There is one more trick involved in practice. If the teacher is too confident, its probability distribution comes out very sharp — something like 98% / 1% / 1% — which is not much different from a hard label, so the advantage of soft labels never shows up.
The fix is to divide by a number T when computing the probabilities, the temperature. The larger T is, the flatter the distribution, and the clearer the relative relationships between words become. This is the same mechanism as the Temperature covered in Part 2, just used for a different purpose: there it was to make output more varied, here it is to pass more information along to the student.
This is the hardest part of the section to picture in the abstract, so here it is to drag directly. Below is the judgment a teacher gives when it looks at a photo of a cat, and T is yours to set.
The pipeline on its own stays abstract. Here is a public example: when DeepSeek released R1 in January 2025, it also open-sourced six distilled models, all trained on reasoning data generated by R1.
One detail in this list deserves attention: the students are not built on DeepSeek's own base models — they were picked from other companies' open-source releases. Four of the six use Qwen2.5, two use Llama3.
DeepSeek never explained the choice, but you can work backwards to a few requirements. The base has to ship open weights under a license that permits further training, or you cannot release what you build. The size range has to be complete enough to cover everything from 1.5B up to 70B in one pass. And the community tooling has to be mature, so training and deployment do not mean rebuilding the wheel. Not many options met all three at the time.
In the evaluations DeepSeek published, the 32B distilled version beat OpenAI o1-mini on several math and code benchmarks, and the company described it as reaching a new best result among the dense models of that period.
That covers most of what distillation buys you: cheap, fast, able to run locally. The next lesson covers the cost. The student learns everything the teacher has, including the teacher's bad habits.
Put “Five Steps, Start to Finish” back into its constraints
“Collect questions that cover the target domain.” shows that a model, license, access route, or leaderboard is information—not an answer outside context. The real choice depends on task, data boundary, latency, quality floor, and operating cost.
Write elimination criteria before chasing the top score
The comparison in “The word that matters is "full." Keep only the final answer and the student just memorizes conclusions;” should use the same real inputs while observing correctness, failure behavior, response time, and cost. A model leading a public leaderboard may still fail your license, privacy, or peak-latency constraints.
Without a test set, there is no reliable winner
Start with “That covers most of what distillation buys you: cheap, fast, able to run locally.”: choose inputs that could genuinely change the decision and write down one counterexample that would reverse your choice. That is more useful than memorizing a single ranking.
From “Five Steps, Start to Finish” to “Why Step 3 Matters”
“Five Steps, Start to Finish” grounds the problem in “Collect questions that cover the target domain. You can curate them by hand or have a model generate them. This step sets the ceiling on what the student can do: whatever domain you never ask about, the student…”. “Why Step 3 Matters” then moves it toward “Say you have an image classification question where the correct answer is cat. Conventional training tells the model: cat is right, everything else is wrong. Distillation hands the teacher's judgment to the stu…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For model selection, write non-negotiable constraints from the real task first. Compare quality, failure behavior, latency, licensing, and cost on the same inputs; use a leaderboard only as a starting point.
- “Five Steps, Start to Finish”: Collect questions that cover the target domain. You can curate them by hand or have a model generate them. This step sets the ceiling on what the student can do: whatever domain you never ask about, the student…
- “Why Step 3 Matters”: Say you have an image classification question where the correct answer is cat. Conventional training tells the model: cat is right, everything else is wrong. Distillation hands the teacher's judgment to the stu…
- “The closing point”: The fix is to divide by a number T when computing the probabilities, the temperature. The larger T is, the flatter the distribution, and the clearer the relative relationships between words become. This is the…
The final “The closing point” brings the discussion to “The fix is to divide by a number T when computing the probabilities, the temperature. The larger T is, the flatter the distribution, and the clearer the relative relationships between words become. This is the…”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.