Quantization without myths: what you gain, what you lose, what to measure
Quantization is what lets a 27B model run on one graphics card and a 120B model on a single server GPU. It isn't free, and the cost rarely shows in benchmark averages. Here is what the measurements say as of September 2026, and how we decide.
Quantization stores a model's weights in fewer bits (8, 5, 4, sometimes 2) instead of 16. Memory falls almost in proportion and, since generation is limited by memory bandwidth, speed rises: Llama 3.1 8B goes from 14.96 GiB and 29 tokens per second in F16 to 4.58 GiB and 72 in Q4_K_M.
The quality cost depends on method, model size and task: 8 bits is practically lossless, 4 to 6 bits a good trade and under 4 bits a risk, above all for small models, agents and non-English text. Our rule: the format your hardware accelerates, native or QAT releases when they exist, and validation in Spanish and French.
What quantization does, in one worked example
A model is billions of weights; in BF16 each takes 16 bits, or 2 bytes. Quantizing stores each weight in fewer bits (at 4 bits, one of 16 possible values) plus a scale shared by each small block of weights. Formats differ in block size, scale storage and which layers keep more bits.
weight memory (GB) ≈ parameters (billions) × bits per weight ÷ 8
8B in BF16: 8 × 16 ÷ 8 = 16 GB
8B in Q4_K_M: 8 × 4.9 ÷ 8 ≈ 4.9 GB
70B in Q4_K_M: 70 × 4.9 ÷ 8 ≈ 43 GBThis matches llama.cpp: Q4_K_M averages 4.89 bits per weight and Llama 3.1 70B lands at 43.1 GB. With one user, reading weights from memory is the bottleneck, so speed follows size: from 29 to 72 tokens per second between F16 and Q4_K_M. What the formula leaves out: the KV cache and quality.
The format map: what runs where
Your engine and hardware choose the format before any leaderboard does.
| Format | Bits per weight | Where it runs | Best for |
|---|---|---|---|
| GGUF k-quants (Q4_K_M, Q5_K_M, Q6_K, Q8_0) | 3 to 8.5 | llama.cpp, Ollama, LM Studio on CPU, Apple Silicon, NVIDIA, AMD | Laptops, workstations |
| GGUF i-quants with imatrix (IQ1_S to IQ4_XS) | 2 to 4.5 | Same engines | Big models in little memory |
| GPTQ, AWQ (W4A16) | 4 plus scales | vLLM, SGLang, Transformers | GPU servers, few users |
| EXL3 | Fractional | ExLlamaV3, TabbyAPI (NVIDIA) | Quality per bit on one consumer GPU |
| bitsandbytes (int8, NF4) | 8 or 4 | Transformers, PEFT | QLoRA fine-tuning |
| FP8 (W8A8) | 8 | vLLM, SGLang on Ada, Hopper, Blackwell, AMD MI300X | Multi-user production |
| NVFP4, MXFP4 | 4.5 and 4.25 | Blackwell; AMD MI355X (MXFP4); llama.cpp reads both | New servers, native FP4 models |
Names are recipes: Q4_K_M is a 4-bit "medium" mix that gives 6 bits to some sensitive tensors, and i-quants with an importance matrix (imatrix) use calibration text to decide where rounding hurts most. On GPUs, the archived AutoGPTQ and AutoAWQ gave way to GPTQModel and vLLM's llm-compressor (v7.5.0 and v0.14.0, September 2026); ExLlamaV3 (v1.5.2) replaced ExLlamaV2, though its server, TabbyAPI, calls itself a hobby project.
How much quality you lose: the evidence
Perplexity says little; divergence from the original and flipped answers say more.
| Type | Size | KLD | Reading |
|---|---|---|---|
| Q8_0 | 7.96 GiB | 0.0014 | Practically lossless |
| Q6_K | 6.14 GiB | 0.0055 | Near lossless |
| Q5_K_M | 5.33 GiB | 0.0108 | Near lossless |
| Q4_K_M (imatrix) | 4.58 GiB | 0.0282 | The usual balance |
| Q3_K_M (imatrix) | 3.74 GiB | 0.0844 | Visible loss |
| Q2_K | 2.96 GiB | 0.4451 | Heavy loss |
| IQ1_S (imatrix) | 1.88 GiB | 2.2546 | Unusable on a small model |
Averages hide damage: in Accuracy is Not All You Need (Microsoft Research, 2024), schemes that moved accuracy by 0% to 2% still flipped up to 13.6% of answers. Size matters too: in the ACL 2025 study Give Me BF16 or Give Me Death?, 4-bit weights kept 99.5% of a 32B reasoning model's score but 93.5% for a 1.5B.
Language matters most for us. Cohere's study of quantized multilingual models (Findings of EMNLP 2024) found that, for a 103B model at 4 bits, automatic metrics showed a 0.3% drop in French while human raters measured 16.6%.
Below 4 bits the curve bends, and Unsloth advises against 1-bit quants for agents: they loop and break tool calls. At equal memory, a bigger model at 4 to 5 bits usually wins: in Unsloth's tests, Gemma 3 27B in Q4_K_M (15.4 GB) scores 71.2% on MMLU versus 67.2% for Gemma 3 12B in BF16 (about 24 GB).
FP8, FP4 and models that ship already quantized
On recent GPUs the format is built into the silicon, and more models are trained for it.
FP8 is the safe choice on servers: the same ACL 2025 study (over 500,000 evaluations on Llama 3.1) found it "effectively lossless" at every size, with INT8 losing 1% to 3%. vLLM cites half the memory and up to 1.6 times the throughput, but FP8 compute needs Ada, Hopper or Blackwell; an A100 runs FP8 checkpoints only as 8-bit weights.
MXFP4 and NVFP4 are 4-bit floating-point formats: MXFP4 shares a power-of-two scale across 32 values (4.25 bits per weight), NVIDIA's NVFP4 a finer FP8 scale every 16 (4.5 bits). NVIDIA reports "1% or less" degradation moving DeepSeek-R1-0528 from FP8 to NVFP4. Only Blackwell accelerates NVFP4; MXFP4 is also accelerated on AMD's MI355X.
Native releases change the rules. OpenAI shipped gpt-oss (August 5, 2025) with MXFP4 expert weights, so the 120b fits one 80 GB GPU, and its published scores are those of the quantized model. Kimi K2 Thinking uses native INT4, DeepSeek-V4 (April 2026) ships FP4 experts, and Gemma 4's quantization-aware training (QAT) release (June 5, 2026) takes the 31B from 69.9 GB to 17.5 GB.
If the vendor publishes a native or QAT low-bit checkpoint, use it as shipped. Re-quantizing adds error: converting DeepSeek-V4-Flash's MXFP4 experts to Q4_K nearly tripled the KLD in Unsloth's measurements (0.029 versus 0.010).
The KV cache: the memory nobody budgets
At long context, the conversation's working memory can outweigh the model.
Each token in context stores keys and values in every attention layer; the size comes from the model's config.json:
KV per token = 2 × layers × KV heads × head dim × bytes per value
Qwen3-32B, 16-bit: 2 × 64 × 8 × 128 × 2 B = 256 KiB
32,768 tokens ≈ 8 GiB, 131,072 tokens ≈ 32 GiB (per request)Architecture changes the result: gpt-oss-120b, with sliding-window layers, needs 36 KiB per token, and Qwen3.8-27B, with linear-attention layers, 64 KiB. Each parallel request carries its own cache, so memory scales with parallel slots times context length, as Ollama warns.
Quantizing the cache is the next lever: q8_0 or q4_0 in Ollama (OLLAMA_KV_CACHE_TYPE) and llama-server (--cache-type-k, --cache-type-v), FP8 in vLLM (--kv-cache-dtype). A COLM 2025 study found a 4-bit cache lossless on 14B and 32B models but a 3-bit cache costing over 5% on 1.5B and 7B, and recommends 8 bits; Unsloth saw 4-bit caches degrade output.
Compute the cache at your real p95 context times the requests you expect at once, and start with q8_0 or FP8 when memory is short. Use q4_0 only after testing your longest real documents.
Which format for which hardware
Our starting points as of September 2026; your own evaluation decides.
| Hardware | Start with | Avoid |
|---|---|---|
| Laptop or PC: 8 to 16 GB GPU or 16 to 32 GB unified memory | 4B to 14B in Q4_K_M or QAT; Q5_K_M or Q6_K if it fits | IQ2 and IQ1 on small models |
| One 24 to 32 GB GPU (RTX 3090, RTX 5090) | 27B to 32B in Q4_K_M or Q5_K_M, or EXL3 at 4 to 5 bits | A 70B squeezed into 2 bits |
| Mac with 64 to 512 GB of unified memory | 4-bit GGUF or MLX; native MXFP4 such as gpt-oss | Expecting GPU-server throughput |
| DGX Spark (128 GB at 273 GB/s) | Native MXFP4 or NVFP4 MoE: gpt-oss-120b at about 61 tokens/s | Dense 70B: 2.7 tokens/s in FP8 |
| Ampere server (A100), few users | W4A16 (AWQ, GPTQ); INT8 W8A8 as concurrency grows | Counting on FP8 compute |
| Ada or Hopper server (H100, H200), many users | FP8 W8A8 plus FP8 KV cache | Untested 4-bit weights under heavy load |
| Blackwell server (B200, B300, RTX PRO 6000) | NVFP4 or FP8; the vendor's FP4 checkpoints | Re-quantizing native FP4 |
The server rows follow the ACL 2025 study: 4 bits pays off when a few users wait, 8 bits wins under continuous batching. On Macs and the DGX Spark bandwidth rules, so mixture-of-experts (MoE) models with few active parameters run far faster. More in local AI hardware in 2026 and from laptop to server.
How to validate a quant on your own tasks
Half a day of testing prevents weeks of silent failures.
- Fix an evaluation set from real work, with expected answers. At Slash we use 30 fixed prompts per client in Spanish, English and French (method in our guide to LLM evaluations).
- Set a reference: the same model in BF16 or FP8, or the native release, with greedy decoding.
- Measure KLD and flipped answers on your own text, per task and per language, not on the calibration text: Unsloth warns that this overstates quality.
- Read Spanish and French by hand: accents, agreement, register (tú or usted, vous), drift into English.
- Test tool calls and JSON with the production chat template, validating each output against its schema.
- Test long context with the production KV cache, and record hash, quant type, engine version and template to re-run everything when something changes.
Five common mistakes
- The wrong chat template. A GGUF carries its own Jinja template, which llama-server applies by default. If it doesn't match the training format, answers degrade with no visible error and tool calls fail first. Templates can also hide instructions (Pillar Security, 2025): review them like code.
- Over-quantizing small models. IQ1_S takes Llama 3 8B's perplexity from 6.2 to 60.7. Below 8B, stay at 4 bits or more.
- Ignoring the default context. Per Ollama's docs, it uses 4K tokens below 24 GiB of VRAM: a long document won't fit until you raise it.
- Re-quantizing what is already quantized: native MXFP4 or INT4 weights, or a GGUF built from a quantized checkpoint rather than BF16. And keep the vision projector (mmproj) in BF16 or Q8, as llama.cpp advises.
- Trusting someone else's number. A perplexity measured on other text with another engine says little about your case. To pick the model first, see open-weight models in 2026.
Key takeaways
- Quantization trades bits for memory and speed: Llama 3.1 8B drops from 14.96 GiB in F16 to 4.58 GiB in Q4_K_M and generates about 2.5 times faster.
- Q8_0 and FP8 are practically lossless; Q4_K_M is the sensible default; below 4 bits quality falls fast.
- Small models, agents, Spanish and French suffer more than averages say: a 16.6% loss in French for human raters, 0.3% on automatic metrics.
- FP8 on Ada, Hopper and Blackwell, FP4 on Blackwell, GGUF elsewhere; use native and QAT releases as shipped.
- Budget the KV cache and validate every quant on your own prompts, JSON and tool calls.
Sources
- llama.cpp quantize tool: README (types, sizes, speeds)
- llama.cpp perplexity tool: README and KLD scoreboards
- "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
- Accuracy is Not All You Need
- How Does Quantization Affect Multilingual LLMs?
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
- Introducing NVFP4 for Efficient and Accurate Low-Precision Inference
- FP8 W8A8 quantization
- gpt-oss-120b model card
- Quantization-aware training (QAT) checkpoints for Gemma 4
- Unsloth Dynamic GGUFs (Dynamic 2.0 and 3.0)
- Ollama FAQ: K/V cache quantization and concurrency
Editorial note: this analysis reflects the public information available on the review date. Models, prices and rules change fast; every third-party figure links to its source, and our opinions are labeled as such. Spotted an error? Write to contact@slash-digital.io.
The questions we hear often
What is the best quantization for Ollama or LM Studio?
Q4_K_M, the default in Ollama's library, is the usual balance. If memory allows, Q5_K_M and Q6_K are near lossless and Q8_0 practically lossless. If the vendor publishes a QAT version, such as Gemma's Q4_0, start with that one.
Does quantization make a model faster?
For one user, yes: generation is limited by memory bandwidth, so smaller files generate faster (29 to 72 tokens per second from F16 to Q4_K_M in llama.cpp's example). On servers with many simultaneous requests, 8-bit formats such as FP8 are often more efficient than 4-bit weights.
Can I run a 70B model on a 24 GB GPU?
Not comfortably. In Q4_K_M a 70B needs about 43 GB for weights alone, so you would have to offload layers to system memory, which is slow, or go below 3 bits, which costs quality. A 27B to 32B model at 4 to 5 bits is usually the better choice.
Is FP8 better than 4-bit AWQ or GPTQ?
They solve different problems. FP8 is practically lossless and suits high-concurrency serving on Ada, Hopper or Blackwell. 4-bit weights compress about 3.5 times and suit latency-bound serving with few users, with a small loss you should measure.
Does quantization hurt Spanish and French more than English?
It can, and automatic benchmarks hide it: in Cohere's study, a 4-bit 103B model lost 0.3% in French on automatic metrics but 16.6% according to human raters. Always include native-speaker review in your validation.
More analysis to read next
Local AI in 2026: which hardware to buy and what runs on it
RTX 5090, RTX PRO 6000, DGX Spark, Ryzen AI Max+ and Mac Studio M5: memory math, what fits where, buy-vs-rent costs and picks by team size.
Local AI and open weightsFrom laptop to server: Ollama, llama.cpp, vLLM and SGLang
Which engine for one person, a team or production, the techniques that multiply throughput, how to secure and monitor it, and when an API is cheaper.
Agents and engineeringEvals: how to test an LLM system
Why eyeballing fails: a golden set in Spanish and French, calibrated LLM judges, CI regression gates, tooling and what running evals costs.
Let's put it in production
Tell us your challenge. We reply within 24 business hours with an honest first read: if we can help, we'll say how; if not, we'll say who can.
I reply personally. No endless forms, no canned replies.