Why Make Models Smaller
Three practical motives — cost, speed, on-premise deployment — and the things small models cannot do
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhy Make Models Smaller?
Three practical motives — cost, speed, on-premise deployment — and the things small models cannot do
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Costthe gap is orders of magnitude, not percentages
Drop the parameter count by an order of magnitude and the compute per inference usually drops by an order of magnitude too. The common industry figure is that inference on a small model can cost one to two orders of magnitude less than on a flagship. Exactly how much less depends on size, quantization level, batch size, and deployment method. Don't memorize the numbers; remember that this is a difference in magnitude, not a discount.
Speedrunning locally removes the whole network round trip
Calling a remote flagship model means a network round trip, then queueing, then generating token by token. Waiting several seconds is normal. A small model running locally cuts out the network leg entirely, so the first token arrives much faster. Conversational products are especially sensitive to this: waiting three seconds and waiting three tenths of a second are two different experiences.
On-premise and compliancesome data is simply not allowed off the internal network
Medical records, case files, internal code, unreleased financial data — for this kind of content the compliance requirement is that it never leaves the internal network. However capable the model on the other end is, it is unusable, because the first step already fails. A small model deployed locally is the only route. The control described back in the first section becomes a hard constraint here.
What does "one to two orders of magnitude" turn into on the invoice at the end of the month? Drag in your own product's request volume and see.
"Small" is a relative term, but the test is concrete: does it fit on the GPU you have. There is an interactive tool later in this chapter dedicated to this calculation; here is the formula it uses.
Use 2.6 for FP16 and 0.65 for INT4. The coefficient already includes runtime overhead such as the KV Cache, so don't multiply it in again afterwards.
Here are a few sizes on that basis. The last column is whether it runs on a 24 GB consumer GPU after INT4 quantization.
The conclusion is blunt. Quantized, the 8B size runs on a gaming GPU from several years ago; flagship sizes will not fit no matter how you quantize them. Which turns the question into a different one: can a small model's capability be brought up to standard.
There are two ways to close the capability gap. One is to have the small model learn from scratch on its own, repeating the whole cycle of data, compute, and trial and error. The other is to find a large model that has already learned and let it teach. The second is knowledge distillation.
The difference from conventional training lies mainly in how much information gets transferred at a time.
Training data provides the correct answer. Right earns credit, wrong loses it. What comes back from one item is basically a single "right" or "wrong." Whether it was wildly wrong or off by a hair is invisible in the label. The signal is sparse, which is why it takes enormous data and enormous compute to make up for it.
The teacher provides not just the final answer but its leanings across all the candidates. On the same item, the student can see which options the teacher considered close and which were nowhere near. The same piece of data carries far more information, so the student learns much faster.
How that information is actually transferred and how the training math works is what oss-6 takes apart. For this section, just hold on to the motivation: distillation saves money and time; it does not create new capability.
On 20 January 2025, DeepSeek released R1 and open-sourced a set of distilled small models alongside it. The approach was to generate reasoning data with R1 and then train smaller models on that data. Some of the distilled versions are built on Llama as the base, others on Qwen.
This one is worth noting because it is a publicly verifiable third-party choice. Which bases a company picks when it does distillation says more than any marketing claim. R1 uses the MIT License, and the official announcement states explicitly that training other models through distillation is permitted. Only because that was written down in plain terms did anyone dare take this route seriously.
The naming convention for that batch of models and how it was done in detail come in the next section.
2. Whether something can ship comes down to VRAM first, and the formula is parameter count times the precision coefficient.
3. Distillation has a large model teach a small one, in order to avoid the cost of training from scratch.
4. Small models have a capability ceiling, and they inherit their teacher's defects. Put both on the table before you decide.
The next section covers how distillation actually works: how the teacher sets the questions, how the student checks its answers, and which engineering pitfalls sit in between.
Put “The short answer” back into its constraints
“Drop the parameter count by an order of magnitude and the compute per inference usually drops by an order of magnitude too.” shows that a model, license, access route, or leaderboard is information—not an answer outside context. The real choice depends on task, data boundary, latency, quality floor, and operating cost.
Write elimination criteria before chasing the top score
The comparison in “Calling a remote flagship model means a network round trip, then queueing, then generating token by token.” should use the same real inputs while observing correctness, failure behavior, response time, and cost. A model leading a public leaderboard may still fail your license, privacy, or peak-latency constraints.
Without a test set, there is no reliable winner
Start with “The next section covers how distillation actually works: how the teacher sets the questions, how the student checks its answers, and which engineering pitfalls sit in between”: choose inputs that could genuinely change the decision and write down one counterexample that would reverse your choice. That is more useful than memorizing a single ranking.
From “The short answer” to “Three reasons to make models smaller”
“The short answer” grounds the problem in “Models get smaller because reality forces it. Flagship models are the most capable, but they are expensive, they are slow, and they require sending your data to somebody else's servers. In many scenarios a smal…”. “Three reasons to make models smaller” then moves it toward “Drop the parameter count by an order of magnitude and the compute per inference usually drops by an order of magnitude too. The common industry figure is that inference on a small model can cost one to two orde…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For model selection, write non-negotiable constraints from the real task first. Compare quality, failure behavior, latency, licensing, and cost on the same inputs; use a leaderboard only as a starting point.
- “The short answer”: Models get smaller because reality forces it. Flagship models are the most capable, but they are expensive, they are slow, and they require sending your data to somebody else's servers. In many scenarios a smal…
- “Three reasons to make models smaller”: Drop the parameter count by an order of magnitude and the compute per inference usually drops by an order of magnitude too. The common industry figure is that inference on a small model can cost one to two orde…
- “The closing point”: There are two ways to close the capability gap. One is to have the small model learn from scratch on its own, repeating the whole cycle of data, compute, and trial and error. The other is to find a large model…
The final “The closing point” brings the discussion to “There are two ways to close the capability gap. One is to have the small model learn from scratch on its own, repeating the whole cycle of data, compute, and trial and error. The other is to find a large model…”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.