---
title: "GGUF quantization, explained: how to pick the right quant"
date: 2026-11-22
description: "What Q4_K_M, Q5_K_M, Q6_K and Q8_0 actually mean, how to estimate VRAM from parameter count, and which quant fits an 8, 16, 24 or 48 GB card."
tags: ["quantization", "gguf", "vram", "ollama", "llama-cpp", "basics"]
---

Most local-model problems start with the wrong file, not the wrong model. A 27B build at Q4_K_M can run comfortably on a 24 GB card; the same model at Q8_0 will not. A 4-bit file is not automatically worse than an 8-bit one for chat, and a 2-bit file is not a free way to run a 70B.

This guide explains what the names mean, how to estimate memory before you download, and which quant to pick for common cards. It applies to GGUF files used by Ollama, LM Studio and llama.cpp.

## What a quant actually changes

A model is trained in 16-bit or bfloat16 precision. Quantization stores the weights with fewer bits per value. The inference engine converts them back to a higher precision for the matrix multiplies, so the model still computes in 16-bit, but it reads far less data from memory.

That gives three effects:

- **Memory:** the file and the resident weights shrink roughly in proportion to bits per weight.
- **Speed:** token generation is usually memory-bandwidth-bound, so smaller weights can be faster.
- **Quality:** fewer bits mean more rounding error. How much that matters depends on the model, the quant recipe and your task.

## Reading the label

Take `Q4_K_M`:

- **Q4** — nominally 4-bit weights.
- **K** — a “k-quant” recipe that mixes precision across tensors instead of using one bit width everywhere.
- **M** — the medium variant. `S` is small, `L` is large.

`Q8_0` is an older-style 8-bit quant. `IQ3_XXS` is an importance-matrix quant: it uses calibration data to decide which weights matter most, which helps at very low bit widths.

| Quant | Rough bits per weight | When it makes sense |
| --- | --- | --- |
| Q8_0 | ~8.5 | Evaluation, small models, or maximum fidelity |
| Q6_K | ~6.6 | You have headroom and want near-lossless chat |
| Q5_K_M | ~5.7 | A good quality step up from Q4 |
| Q4_K_M | ~4.8 | The default for most local chat |
| Q4_K_S | ~4.6 | Slightly smaller Q4, slightly rougher |
| Q4_0 | ~4.5 | Legacy; prefer a K-quant when one exists |
| Q3_K_M | ~3.9 | Tight VRAM; expect visible degradation |
| IQ3_XXS | ~3.1 | Last resort before dropping a model size |
| Q2_K | ~2.6 | Only for experiments or extreme constraints |

The bits-per-weight figures are approximate. File size also includes metadata and the tokenizer, and different models quantize differently.

## Do the memory math

A rough weight estimate:

```
weights in GB ≈ parameters in billions × bits per weight ÷ 8
```

| Model | Q4_K_M (~4.8 bpw) | Q8_0 (~8.5 bpw) |
| --- | --- | --- |
| 4B | ~2.4 GB | ~4.3 GB |
| 8B | ~4.8 GB | ~8.5 GB |
| 12B | ~7.2 GB | ~12.8 GB |
| 24B | ~14.4 GB | ~25.5 GB |
| 27B | ~16.2 GB | ~28.7 GB |
| 35B | ~21 GB | ~37 GB |
| 70B | ~42 GB | ~74 GB |
| 122B | ~73 GB | ~130 GB |

Then add runtime overhead: the context/KV cache, the compute buffer, and a little slack for the backend. A safe starting point is the weight estimate plus 1–3 GB, more for long context.

Mixture-of-experts models need a separate note. A 35B-A3B model has 35B total parameters but activates only a few billion per token. It still needs all experts resident, so budget memory from the **total** parameter count; the small active count mainly buys speed.

## Context is part of the budget

The KV cache grows with context length. For a 27B model with grouped-query attention, 8K tokens of FP16 cache can cost around 2 GB; 32K can add 8 GB or more. Long-context chats fail on memory even when the weights fit.

Ways to buy room back:

- **Lower the context** to what you actually use. 8K is enough for most chat and coding turns.
- **Quantize the KV cache** (`q8_0` is a common choice) in llama.cpp or a compatible backend.
- **Enable flash attention**, which reduces the memory spent on attention.
- **Use a smaller model** rather than a lower-bit quant of a large one.

## Pick by card

These are starting points for a single GPU, with room for 8K context:

- **6–8 GB:** a 4B model at Q4/Q5, or a 7–8B at Q4 with a short context.
- **12 GB:** a 12–14B at Q4_K_M, or a 7–8B at Q6/Q8.
- **16 GB:** a 14B at Q5_K_M/Q6_K, a 24B-A2B MoE at Q4, or a 27B at Q3/IQ3 if you accept the quality hit.
- **24 GB:** a 27B at Q4_K_M or Q5_K_M, a 35B-A3B MoE at Q4, or a 12B at Q8.
- **48 GB:** a 70B at Q4_K_M, a 35B-A3B at Q6, or a 27B at Q8 with a long context.
- **Multi-GPU / 96 GB+:** 122B-class MoE models at Q4, or 70B at Q6/Q8.

The catalog’s `approxVRAM` field uses Q4 as the reference. Treat it as a floor, not a promise.

## Quality: what you actually lose

For most chat, coding and writing tasks, **Q4_K_M is the sweet spot**. It is close enough to the base model that other factors — prompt, sampler, context — matter more.

- **Q5_K_M and Q6_K** are worth it when the model fits with room to spare, especially for small models where the absolute quality gap is easier to notice.
- **Q8_0** is useful for evaluation, for small models where the memory cost is trivial, or when you want to rule quantization out as a variable.
- **Q3 and below** start to change behavior: more formatting slips, weaker instruction following, and worse performance on multi-step reasoning. Use them to fit a model you could not otherwise run, not as a default.
- **IQ quants** can be better than plain Q3/Q2 at the same size, but they may be slower on some hardware and depend on an importance matrix. Check that your backend supports the file.

Quantization is not the only source of behavior change. A fine-tune, a different chat template, or a sampler setting can move output more than one quant step. Test on your own prompts instead of trusting a single benchmark table.

## Run it

With Ollama, append the quant to a Hugging Face GGUF repo:

```bash
ollama run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M
```

With a recent llama.cpp build, use `-hf` and offload the layers you can fit:

```bash
llama-cli -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:Q4_K_M \
  -c 8192 -ngl 99
```

In LM Studio, search for the repo, open the download list, and pick the same quant tag. The model card usually lists every available file.

If a tag does not exist, the repository may name quants differently. Open the Files tab on Hugging Face and match the exact filename.

## Rules of thumb

1. Start at **Q4_K_M** unless you have a reason not to.
2. Estimate weights from total parameters, not active parameters.
3. Add 1–3 GB for overhead, then check your context budget.
4. Move up to Q5/Q6 before moving down to Q3.
5. Test the quant on your real prompts; do not assume one step is invisible.
6. Prefer a smaller model at Q4 over a larger model at Q2.

## Next steps

- Browse the [catalog](/models) and compare Q4 memory estimates.
- Read [What is Abliteration?](/guides/what-is-abliteration) if you are choosing a method, not just a file.
- Follow the [Ollama quickstart](/guides/run-qwen3-8-27b-abliterated-with-ollama) for a full setup with sampling defaults.

---

Canonical: https://www.wahloon.net/guides/gguf-quantization-guide

- Older: [Quickstart: Run Huihui Qwen3.8-27B-abliterated with Ollama / LM Studio](https://www.wahloon.net/guides/run-qwen3-8-27b-abliterated-with-ollama)
