Quantization in plain English

Why does one model come in several file sizes?

Because the files can be shrunk after the model is built, and people shrink them by different amounts. LM Studio's download guide says you'll often see options with names like Q3_K_S and Q_8. They're all copies of the same model, provided in varying degrees of fidelity.¹

That Q stands for quantization.

What is quantization?

A model is mostly a huge list of numbers. Each number takes up room, and how much depends on how precisely it's stored. Quantization stores each one less precisely.

Hugging Face describes it as going from a high-precision representation, usually 32-bit floating point, to a lower precision data type.² Think of rounding prices to the nearest dollar instead of keeping the cents. The totals come out close, not identical.

The payoff is size. Hugging Face says reducing the number of bits means the resulting model requires less memory storage.²

What gets lost?

Some precision. LM Studio says quantization roughly means compressing model files in size while giving up some degree of quality.¹

Wikipedia is a little more hopeful about the aim. It says post-training quantization tries to cut the space needed while preserving most of the model's performance.³ But it also warns that the simplest approach, cutting every parameter to a set number of bits, loses much precision.³

So the result depends on the method and on how far you squeeze. Nobody can promise you that a smaller file answers just as well for your work.

How many bits should I pick?

LM Studio's advice is short: choose a 4-bit option or higher if your machine is capable enough for running it.¹

Ollama's FAQ has a figure that is easy to misread. Its 1/4 figure applies only to an optional setting for the K/V context cache, the memory used for the conversation so far. It does not apply to the model download. For that setting, the FAQ says the 4-bit choice uses approximately 1/4 the memory of f16. It adds that the loss in precision is small-medium and may be more noticeable at higher context sizes.⁴

That cuts two ways. A lower number saves memory. A higher number keeps more detail.

How do I choose in practice?

1. Work out how much memory you can spare, using the hardware page.
2. Pick the biggest file that fits comfortably, but stay at four bits or higher if you can, as LM Studio suggests.¹
3. Try it on a few real, non-confidential questions from your own work.
4. If answers look sloppy, try a larger file or a smaller model.

The sums for each size are on the model sizes page.

Is a quantized model a different model?

LM Studio calls the options copies of the same model.¹ Still, read the licence that comes with the exact file you download. The licence page explains what to look for.