Model sizes and memory explained / Local Model Notes

What does 7B or 70B mean?

It's a count of parameters. A parameter is one of the numbers inside the model that was set during training. Wikipedia says the count is usually written as nM for millions, nB for billions or nT for trillions.¹

So a 7B model has about seven billion of them, and a 70B model has about seventy billion.¹ More parameters usually means a bigger file and more memory.

Work out the file size first, then leave room for the working notes.

Do more parameters mean a better answer?

Not by itself, and this page won't rank models. What you can count on is the cost. A bigger model needs more memory and often runs slower on the same computer.

Wikipedia says the largest models typically have more than 100 billion parameters, which places them outside the range of most consumer electronics.¹

How much memory does a size need?

Start with the arithmetic. Wikipedia explains that one float16 number has 16 bits, or 2 bytes, so one billion parameters require 2 gigabytes.¹

That is the float16 figure Wikipedia gives.¹ For example:

A 7B model stored this way comes to about 14 gigabytes.¹
A 70B model comes to about 140 gigabytes.¹

A full-precision float32 copy would be larger still. Many downloads are smaller than the float16 figure because they are shrunk first, which is what quantization means. Ollama's FAQ has an ollama ps example that shows a llama3:70b model loaded in memory at 42 GB.² That is the size in memory, not the download size. It's one example, so check the size shown on the page of the model you want.

For comparison, the example model in Ollama's quickstart has a download of about 7.2 GB.³ The quickstart doesn't give its parameter count here, so treat it as one data point only.

What else uses memory besides the model?

The conversation does. A model reads your question plus earlier text, up to a limit called the context window. Ollama says its default is 4096 tokens.² A token is a chunk of text, often a word or part of one.

Ollama's quickstart says larger context windows need more memory.³ So a long document plus a big model can run out of room even when the file itself fit.

LM Studio's requirements page puts it plainly: LLMs can consume a lot of RAM.⁴

What does a name like 320B-A18B mean?

Some models are built as a "mixture of experts". Wikipedia says only a fraction of the parameters are used for each input.¹ It gives "320B-A18B" as an example of a label with two numbers: the total, then the count active at a time.¹

That can change how much work each answer takes. It doesn't tell you how much memory to set aside, so read the model's own download page for that.