"Will it fit on our hardware?" is the wrong first question. It's the one everyone asks, because it's free to answer — the thing either loads or it doesn't.
Throughput costs you a measurement. So the capacity gate passes, and it feels like the decision is made.
Mac Mini M4, 24GB unified memory, ~120GB/s memory bandwidth. A 27B model, IQ4_XS quantized, 15GB on disk.
Capacity gate: pass. Metal's recommendedMaxWorkingSet is 17.76GB, the model is 15GB, ollama ps reports 100% GPU resident. No swap, no spillover. By every "does it fit" criterion this is a clean win.
Generation: 5.6 tokens/second.
That's not a usable interactive worker. It's barely a usable batch worker. And nothing about the capacity check hinted at it.
Autoregressive generation reads the entire model's weights once per token. So:
ceiling ≈ memory bandwidth ÷ bytes touched per operation
= 120 GB/s ÷ 15 GB
= 8 tokens/secondMeasured 5.6 against a ceiling of 8. Ratio 0.70.
That ratio is the whole verdict. When measured throughput is a large fraction of the arithmetic ceiling, you are bandwidth-bound, and you now know something concrete: the bottleneck is not your configuration, not memory pressure, not thermal throttling. It's how fast bytes move.
Rule of thumb I now use: ratio ≥ 0.5 → bandwidth-bound, and size-reduction fixes are dead.
The natural move when capacity is tight is to shrink. Lower quantization, smaller batch, heavier compression. It's the reflex, and in a bandwidth-bound regime it's close to useless.
I was considering Q3_K_M at 13.8GB. Run the same division:
120 ÷ 13.8 = 8.7 tokens/second (up from 8)Under 9% more throughput. For a real drop in output quality, because quantization error doesn't scale linearly with size the way bandwidth does — you give up more than you get, every time, in this regime.
I killed that plan without downloading anything. That's the saving this rule buys: an arithmetic rejection instead of an afternoon of benchmarking a model that couldn't have won.
The deeper reason is that capacity and throughput are governed by different resources. Capacity is bytes of memory. Throughput is bytes per second across a bus. Pulling the capacity lever moves the capacity number. It touches the throughput ceiling only through the incidental fact that a smaller model has fewer bytes to stream — a weak, strictly linear coupling, and you pay for it non-linearly.
This is the part that makes the arithmetic worth doing. Once you know it's bandwidth, the same model on different hardware is a division away:
| Machine | Bandwidth | Ceiling for a 15GB model |
|---|---|---|
| M4 | ~120 GB/s | 8 tok/s |
| M4 Pro | ~273 GB/s | 18 tok/s |
| M4 Max | ~546 GB/s | 36 tok/s |
The verdict flips on hardware, not on model size. "This model is too slow" was never true — "this model is too slow on 120GB/s" was. Those lead to completely different purchase decisions, and only one of them is correct.
When you correctly name the constrained resource, the set of interventions that can possibly work falls out of it. Naming the wrong one sends you tuning things that were never the problem.
entirely unknown.
coupled and quality degrades faster. This is the classic symptom of conflating the two gates.
resident. Otherwise you're mixing the ceiling with transient congestion — and if you go clean up applications before separating them, you'll never find the actual cause. Here there was no pressure at all and the ceiling was exactly where the arithmetic put it.
Effective is typically 60–80% of theoretical. Include that and the estimate is still good to an order of magnitude — and order of magnitude is the entire decision. 8 vs 5.6 is the same answer. 8 vs 36 is a different one.
1. Before checking capacity, compute the throughput ceiling: ceiling ≈ bandwidth ÷ bytes touched per operation. 2. Measure at least once warm, so cold-load and swap pressure don't contaminate the number. 3. If measured / ceiling ≥ 0.5, declare bandwidth-bound and reject size-reduction fixes. 4. Choose from what's actually left: (a) restrict to latency-tolerant uses, (b) change hardware, (c) move the work off this machine. 5. Write the falsifier as a bandwidth number. If you've named the constrained resource, the point at which that resource changes is the condition that overturns your verdict.
That last step is why this generalizes past local models. The same shape applies to any streaming bottleneck — disk-bound ETL, network-bound sync, cache-line-bound inner loops. "It fits" and "it's fast enough" are separate gates, and the first one is free to check, which is exactly why it gets mistaken for the second.
Bandwidth is the named bottleneck, so the falsifier is a bandwidth number: on a 273GB/s machine this model clears 18 tok/s and the "not usable as a live worker" verdict is void. Any architecture change that stops reading all weights per token — heavy MoE sparsity, aggressive speculative decoding — also breaks the bytes touched = model size assumption the ceiling is built on, and the division has to be redone with the real figure.
Probably not. Compute the ceiling as memory bandwidth divided by bytes touched per operation — for autoregressive generation that is roughly the model size, since every token reads all the weights. If measured throughput is half the ceiling or more you are bandwidth-bound, and swap or memory pressure is not the cause. Measure warm at least once so cold load does not contaminate the number.
In a bandwidth-bound regime, barely. Going from 15GB to 13.8GB moves a 120GB/s ceiling from 8 to 8.7 tokens per second, under 9%, while quantization error grows faster than linearly. Size and speed are only linearly coupled here, so you consistently give up more quality than you gain in throughput.
Divide. The same 15GB model has a ceiling of 8 tokens per second at 120GB/s, 18 at 273GB/s, and 36 at 546GB/s. If the verdict flips on bandwidth rather than on model size, hardware is the lever and shrinking the model is not. Naming the constrained resource correctly is what tells you which interventions can possibly work.