Quantization (GPTQ, AWQ, GGUF)

For NAO + anyone who wants to run a real LLM on a laptop, a single GPU, or a CPU-only box. A 70-billion-parameter model in its native 16-bit weights needs ~140 GB just to hold the weights — more than any consumer GPU has. Quantization shrinks each weight from 16 bits down to 4 (sometimes 3, sometimes 2), turning that 140 GB into ~40 GB or less, with surprisingly small quality loss. This is the single technique that put usable LLMs on hardware you actually own. Siblings: [[lora-and-peft]] (QLoRA fine-tunes on top of 4-bit weights), [[model-routing]] (when to even reach for a big local model), [[attention-and-transformers]] (the layers we're quantizing), [[deepseek-architecture]] + [[mixture-of-experts]] (huge models that need this to run at all).

The one-sentence idea

Store each weight using fewer bits, and choose those bits so the model's *outputs* barely change — not so the weights themselves barely change. Almost every clever quantization method is a different answer to "fewer bits, but which rounding minimizes the output error?"

What it is (intuition first)

A neural network is, at heart, a giant pile of numbers — the weights. After training, each weight is usually a 16-bit floating-point number (FP16 or BF16): 16 bits, two bytes, per weight. A 7B model is 7 billion of those = ~14 GB. A 70B model = ~140 GB.

Quantization asks: do we really need 16 bits of precision per weight to get a good answer? It turns out: no, not even close. You can map the weights onto a much coarser grid — 256 levels (8-bit / int8), or just 16 levels (4-bit / int4), or even 4 levels (2-bit) — and the model still works almost as well.

The intuition for why it works: a trained network is robust. Each individual weight is a tiny vote in a sum over thousands of weights. Nudging any one weight to the nearest allowed value introduces a little noise, but those nudges partly cancel, and the network was never relying on the 11th decimal place of any single weight to begin with. It's like printing a photo: a 24-bit-per-pixel image and an 8-bit-per-pixel image look nearly identical to your eye, because human vision (like a trained network) doesn't depend on perfect color precision.

The simplest scheme — round-to-nearest (RTN) — is exactly the photo analogy:

1. Find the range of weights in a block (say min −0.8 to max +0.9).
2. Carve that range into N evenly spaced bins (N = 16 for 4-bit).
3. Snap every weight to its nearest bin, and remember which bin (4 bits) plus the block's scale and zero-point so you can reconstruct an approximation later.

Reconstruction: w_approx = scale * (q − zero_point), where q is the stored small integer. That's it — that's the whole primitive.

The catch: naive RTN at 4 bits noticeably degrades large LLMs. The interesting methods — GPTQ, AWQ, GGUF's k-quants — are all ways of being smarter than round-to-nearest so that 4-bit (or even 3-bit) stays usable.

Why it matters

Three reasons, in order of how much you'll feel them:

1. Memory — the binding constraint. A weight is loaded once and read every token. If your model doesn't fit in VRAM (or RAM), you either can't run it or you spill to disk/CPU and it crawls. Going FP16 → int4 is a ~4× memory cut. That's the difference between "70B needs 2× A100 80GB" and "70B fits in a single 48 GB card" (or a 64 GB Mac). On the blal.de↗ box — 62 GB RAM, no GPU — quantization is the only reason a local LLM is on the table at all.

1. Speed — LLM inference is memory-bandwidth-bound. Generating one token means streaming every weight from memory through the compute units once. The bottleneck is usually moving the weights, not the math. Quarter the bytes → roughly quarter the memory traffic → big speedups even when the matmul itself stays in higher precision. GPTQ reports ~3.25× on A100, ~4.5× on A6000 vs FP16 (GPTQ, arXiv 2210.17323↗); AWQ's TinyChat reports >3× over HuggingFace FP16 on desktop and mobile GPUs (AWQ, arXiv 2306.00978↗).

1. Reach — it democratizes the models. GGUF + llama.cpp is the reason a Llama-3 8B runs on a Raspberry Pi, an old ThinkPad, or a phone. No CUDA, no datacenter, no cloud bill. This is the substrate the entire local-LLM hobbyist and on-prem ecosystem is built on.

The whole game is the accuracy ⇄ size/speed tradeoff — see the numbers below.

How it works (the real mechanics)

The shared objective: minimize output error, not weight error

Every serious method quantizes one linear layer at a time and frames it as layer-wise reconstruction. Given a layer's weight matrix W and a small batch of real activations X (the calibration data), find quantized weights Ŵ that keep the layer's output as close as possible:

argmin_Ŵ  ‖ W·X − Ŵ·X ‖₂²

(GPTQ, arXiv 2210.17323↗.) This is the key conceptual move. RTN minimizes ‖W − Ŵ‖ (weight error). GPTQ/AWQ minimize ‖WX − ŴX‖ (output error). The difference: some weights matter much more to the output than others (because the activations that multiply them are large), and you should spend your precision budget where it changes the output most.

GPTQ — second-order, error-correcting rounding

GPTQ ("Generative Pre-trained Transformer Quantization") descends from a classic idea, Optimal Brain Quantization (OBQ), itself a quantization twist on Optimal Brain Surgeon. The story:

• Process each weight matrix one column at a time, left to right.
• When you quantize a column, you introduce an error. Don't waste it — push the remaining (not-yet-quantized) columns slightly off their original values to *compensate* for that error. By the time you reach the last column, it has absorbed corrections for every mistake made earlier.

The math runs on the Hessian of the reconstruction loss, which for this layer-wise objective is simply:

H = 2·X·Xᵀ

— a curvature matrix built entirely from the calibration activations X. It tells you how sensitive the output is to each weight and how weights are correlated, so you know how to redistribute error.

Which weight costs the least to quantize (OBQ's greedy pick):

w_q = argmin_{w_q}  (quant(w_q) − w_q)² / [H⁻¹]_qq

The numerator is the raw rounding error; dividing by the inverse-Hessian diagonal [H⁻¹]_qq weights it by curvature — quantize the cheapest weight first.

The compensating update applied to all remaining weights after quantizing q:

δ_F = −  (w_q − quant(w_q)) / [H⁻¹]_qq  ·  (H⁻¹)_{:,q}

Read it as: take the rounding error (w_q − quant(w_q)), normalize by curvature, and smear a correction across the other weights proportional to their correlation with `q` (the q-th column of H⁻¹). All four equations above are verbatim from the GPTQ paper↗.

GPTQ's three engineering tricks that made this tractable at 175B scale:

• Fixed quantization order (don't re-pick the cheapest weight every step — just go left-to-right; empirically the order barely matters at LLM scale). An act-order / desc_act variant does sort columns by activation magnitude for a small accuracy bump.
• Lazy batched updates — apply the error correction to a block of B (≈128) columns at a time instead of after every single weight, so you do big matrix operations instead of tiny ones. This is what turns it from "days" into "hours."
• Cholesky reformulation — naively recomputing H⁻¹ repeatedly is numerically unstable. GPTQ precomputes a Cholesky factorization of H⁻¹ once (with a small damping term added to the diagonal of H — typically ~1% of the mean diagonal — to keep it invertible) and reads the needed columns off the factor.

Result: 175B model quantized to 3–4 bits in ~4 GPU-hours, near-FP16 accuracy (GPTQ↗). GPTQ needs a GPU and calibration data, and uses per-group scales (e.g. group_size=128: one scale per 128 weights) to keep 4-bit sharp.

Footnote for the curious: a 2025 result reframes GPTQ as Babai's nearest-plane algorithm from lattice theory — the back-to-front error correction is exactly Babai rounding against a lattice basis derived from H (arXiv 2507.18553↗). Same algorithm, deeper why.

AWQ — protect the salient weights by scaling, no backprop

AWQ ("Activation-aware Weight Quantization") starts from a sharp observation: not all weights are equally important — and you find the important ones by looking at the activations, not the weights. (AWQ, arXiv 2306.00978↗.)

The findings, in order:

1. Protecting just ~1% of weights (keeping them in FP16) recovers almost all the lost accuracy. So importance is extremely concentrated.
2. The "salient" weights aren't the ones with big weight values — they're the ones multiplied by big activation values (large s_X, the average activation magnitude per input channel). Those are the weights whose rounding error gets amplified into a large output error.
3. But mixed-precision (some weights FP16, most int4) is hardware-ugly — irregular memory layout kills the speedup.

AWQ's trick avoids mixed precision entirely with an equivalent transformation. For a salient channel, scale the weight up by `s` before quantizing and scale the matching activation down by `1/s` — the product is unchanged, but the now-larger weight uses more of the quantization grid, so its relative rounding error shrinks:

Q(w)·x      →      Q(w·s)·(x / s)      with s > 1 for salient channels

The per-channel scale s is found by grid search over a single exponent α, using the activation statistics as the saliency proxy:

s = s_X^α ,     α* = argmin_α  ‖ Q(W·diag(s))·(diag(s)⁻¹·X) − W·X ‖

with α ∈ [0, 1] (α=0 → no scaling = plain RTN; α=1 → full activation scaling). One scalar swept on a small calibration set per layer (Lei Mao's AWQ walkthrough↗).

How AWQ differs from GPTQ:

• No backprop, no Hessian, no error-feedback loop — just statistics + a 1-D search. Faster to run, and it generalizes better to instruction-tuned and multimodal models (the paper claims a first there) because it doesn't overfit the calibration set's reconstruction.
• Both need calibration data; both hit ~4-bit near-lossless. In practice AWQ often edges GPTQ on instruction-following and is the common default in vLLM / TensorRT-LLM serving stacks. GPTQ's error-correction can win on raw perplexity.

GGUF + llama.cpp — k-quants for the CPU/edge world

GGUF is a file format (the successor to GGML), and k-quants are its quantization scheme — this is a different lineage from GPTQ/AWQ, optimized for CPU and mixed CPU/GPU inference in llama.cpp↗.

The mechanics are about block layout, not Hessians:

• Weights are grouped into super-blocks of 256, subdivided into smaller sub-blocks (e.g. 16 or 32 weights).
• Each sub-block gets its own tiny scale (and min), and those per-sub-block scales are themselves quantized (e.g. to 6 bits) against a super-block scale. This two-level scaling is the "k" trick — far better fidelity than one scale per big block, at a tiny bookkeeping cost.
• Mixed precision by tensor: the most sensitive tensors (token embeddings, output/lm_head, sometimes attention) are kept at higher bit-width while the bulk feed-forward weights go low. The _S / _M / _L suffixes encode how aggressive this mix is — Small, Medium, Large.

The naming you'll see, decoded:

• Q4_K_M = 4-bit, K-quant, Medium mix. The community default sweet spot.
• Q5_K_M = 5-bit, a bit safer. Q6_K = near-lossless. Q8_0 = 8-bit, basically a free lunch on quality. Q3_K_* / Q2_K = aggressive, for when memory is desperate. Q4_0 = the old simple (non-k) 4-bit; _K variants beat it.
• `imatrix` ("importance matrix"): an optional calibration pass that, like AWQ/GPTQ, weights the quantization by which weights matter most to real text — it noticeably rescues the low-bit quants (Q2/Q3) and is now standard for good GGUF releases.

llama.cpp's job is then to run these formats fast on a CPU (AVX2/AVX-512/NEON kernels that dequantize-and-multiply on the fly) and to offload N layers to GPU when one is available — the reason it runs everywhere.

The numbers (Llama-3.1-8B-Instruct, llama.cpp)

Concrete measurements from a 2026 unified evaluation (arXiv 2601.14277↗). FP16 baseline = 15,317 MiB, WikiText-2 perplexity 7.32.

The shape of the tradeoff is the whole story: down to ~4–5 bits the loss is fractions of a percent; below ~3 bits it falls off a cliff. The paper flags Q3_K_S (most aggressive 3-bit) as collapsing — GSM8K reasoning dropped 9.3 points (77.6 → 68.3). Practical rules of thumb that fall out of this:

• Q4_K_M / Q4_K_S — the default. ~4× smaller, ~0.5% loss. Use unless you have a reason not to.
• Q5_K_M / Q6_K — when you have the RAM and want near-lossless.
• Q8_0 — quality basically free; use when memory isn't the constraint.
• Q3 / Q2 — only when desperate, and only with an imatrix. Expect reasoning and math to degrade first (they're the most precision-sensitive tasks).

Key ideas & tradeoffs

• Weight-only vs activation quantization. GPTQ/AWQ/GGUF here all quantize weights only — activations stay FP16 and the matmul runs in higher precision (the bottleneck was loading weights anyway). Quantizing activations too (e.g. W8A8, SmoothQuant, FP8) is a separate, harder problem because activations have wild outliers that wreck a naive int8 cast — and it's where most of the remaining compute speedup (not just memory) lives.
• The outlier problem is the central villain. A handful of activation/feature dimensions in LLMs have huge magnitudes; they blow up the quantization range and starve everyone else of precision. AWQ's scaling, GGUF's per-sub-block scales, GPTQ's error feedback, and SmoothQuant's activation→weight migration are all different ways of taming outliers.
• Calibration data matters. GPTQ, AWQ, and imatrix-GGUF all peek at a few hundred real text samples. Pick unrepresentative calibration text and you can quietly overfit — the model looks fine on WikiText and degrades on your domain.
• PTQ vs QAT. Everything here is Post-Training Quantization — quantize an already-trained model, cheap, minutes-to-hours. Quantization-Aware Training (simulate quantization during training) reaches lower bit-widths better but costs a full training run; rare for big LLMs, common for tiny edge models.
• Bits aren't the only axis. Q4_K_M (4.5 bpw mixed) beats Q4_0 (4.0 bpw uniform) at more total bits but better placement of those bits. "How many bits" matters less than "which weights get them."
• Quantize-then-LoRA = QLoRA. You can fine-tune on top of frozen 4-bit weights — see [[lora-and-peft]]. The base stays quantized (small), only the thin adapter trains in higher precision.

Honest caveats & open questions

• "Near-lossless" is benchmark-near-lossless. A 0.5% average drop hides the fact that quantization degrades unevenly: long-context retrieval, multi-step math/reasoning, and rare-token handling tend to suffer first and more than the aggregate suggests. Always eval on your task, not just MMLU/perplexity.
• Perplexity is a weak proxy. Two quants with the same perplexity can behave very differently on instruction-following or code. The community over-indexes on perplexity because it's cheap; trust task evals.
• GPTQ vs AWQ "which is better" has no clean answer. It flips by model, task, group size, and act-order. Treat published win/loss tables as directional, not gospel; benchmark both on the actual deployment.
• Sub-4-bit is still an open frontier. 2-bit and ternary "work" in papers but reliably lose reasoning ability. Whether a robust, general 2-bit LLM is possible (vs. a 2-bit model that happens to pass benchmarks) is unsettled. Newer entrants — GPTVQ (vector quantization), QuIP#, and the asymmetric-calib GPTAQ (arXiv 2504.02692↗) — keep pushing here.
• Quantization can change *safety/alignment* behavior. Compressing weights can subtly shift refusal boundaries and factuality — re-run safety evals after quantizing, don't assume the FP16 alignment carries over unchanged.
• No GPU ≠ no cost. GGUF on CPU is runnable, not fast — token rates on a CPU-only box are a fraction of GPU rates. "Runs on my laptop" and "serves production traffic" are different bars.

How it connects to OpenAlice

Honest, specific ties (skipping where there's no real link):

• The blal.de box has no GPU (62 GB RAM, Ryzen 5 3600 — see global memory). Any local LLM in the ecosystem — for offline eval, a fallback brain, or privacy-sensitive work — is a GGUF + llama.cpp story by necessity. Q4_K_M / Q5_K_M on CPU is the realistic envelope; a 70B model is borderline even quantized, an 8B–14B is comfortable. This is the practical reason the article leans hard on the GGUF section.
• Alice's default brain is hosted (Codex/gpt-5.x via OAuth), not a local quantized model — so quantization is not on Alice's main inference path today. The honest framing: it's the enabling tech for any self-hosted / air-gapped / cost-floor variant, and for the bench platform (bench.blal.pro) if it ever runs local models head-to-head against the hosted harness.
• QLoRA bridge. If OpenAlice ever fine-tunes a small open model (persona specialization, a domain ranker), the realistic recipe is 4-bit base + LoRA adapter — quantization (this article) and [[lora-and-peft]] are the two halves of that single workflow.
• Model routing. Quantized local models are the cheap, private tier in a [[model-routing]] / [[fusion-and-llm-councils]] setup — fast/free for easy turns, escalate to the hosted frontier model for hard ones.

See also

• [[lora-and-peft]] — QLoRA fine-tunes on top of 4-bit quantized weights.
• [[attention-and-transformers]] — the linear layers being quantized.
• [[flash-attention]] — the other main inference-efficiency lever (memory I/O on attention, not weight bits).
• [[model-routing]] · [[fusion-and-llm-councils]] — where a cheap local quantized model fits in a multi-model system.
• [[deepseek-architecture]] · [[mixture-of-experts]] — huge models that practically require quantization (and MoE) to run.
• [[llm-from-scratch]] · [[microgpt-build-an-llm-from-scratch]] — how the FP16 model we're compressing came to exist.