LoRA & Parameter-Efficient Fine-Tuning (PEFT)

For NAO + anyone who wants to specialize a big model without a big budget. Full fine-tuning a 7B model means computing and storing gradients + optimizer state for all seven billion weights — tens of gigabytes of GPU memory, a fresh multi-gigabyte checkpoint per task. LoRA (Low-Rank Adaptation) lets you get ~the same quality by training a tiny set of extra weights — often <1% of the model — while the original weights sit frozen. QLoRA then squeezes the frozen weights to 4 bits so you can fine-tune a 65B model on a single 48 GB GPU. This article is the math, the mechanics, and the honest tradeoffs. Sibling: [[quantization]] (how the 4-bit part works), [[attention-and-transformers]] (what we're adapting), [[llm-from-scratch]] (how the base model got there in the first place).

The one-sentence idea

Don't relearn the whole weight matrix — learn a small, low-rank *correction* to it. Instead of updating a giant weight matrix W, freeze W and learn a thin add-on ΔW = BA built from two skinny matrices. Because real fine-tuning updates turn out to live in a low-dimensional subspace, this skinny correction is enough.

What it is (intuition first)

Imagine a pretrained model as a finished sculpture. Full fine-tuning hands you a chisel and lets you re-carve the entire statue for your new task — powerful, but you need a whole new block of marble (a full checkpoint) and a lot of effort (GPU memory) for every variation you want.

LoRA says: leave the statue alone, and instead bolt on a small, custom attachment. The original sculpture (the frozen base weights) is never touched. You only craft the little attachment — and the attachment is deliberately simple (low-rank), so it has very few moving parts to learn.

The payoff is threefold:

1. Memory. You only store optimizer state + gradients for the small attachment, not the whole model. The original LoRA paper reports a ~3× reduction in GPU memory vs. full fine-tuning of GPT-3 175B, and ~10,000× fewer trainable parameters (Hu et al., 2021, [arXiv:2106.09685]).
2. Storage / shipping. A LoRA "adapter" for a 7B model is often just a few megabytes. You can keep one frozen base model and a library of swappable adapters — one per customer, persona, or task — instead of N full copies.
3. No inference tax. Unlike older "adapter layer" methods that add extra layers (and thus latency), a trained LoRA adapter can be merged back into W after training (W' = W + BA), so deployed inference runs at the exact speed of the original model. This "no additional inference latency" property is one of LoRA's headline advantages over prior adapters (Hu et al., 2021).

PEFT (Parameter-Efficient Fine-Tuning) is the umbrella term for this whole family — LoRA is the most popular member, but it sits alongside prompt tuning, prefix tuning, adapters, IA³, and more (HuggingFace peft library). LoRA won the popularity contest because it's simple, mergeable, and competitive in quality.

Why it matters

• Democratization. Before LoRA/QLoRA, fine-tuning a large model needed a multi-GPU cluster. QLoRA collapsed that to one consumer-ish GPU — the paper's Guanaco models were trained on a single GPU in 24 hours and reached 99.3% of ChatGPT's level on the Vicuna benchmark (Dettmers et al., 2023, [arXiv:2305.14314]). That is the moment fine-tuning became something a small team — or a single engineer — could actually do.
• Many specializations, one base. A frozen base + N tiny adapters is the architecture behind "an adapter per customer / per skill / per persona." You hot-swap behavior without N full models in memory.
• It composes. Adapters can be merged, weighted, added, and subtracted (peft supports linear, cat, ties, dare, svd combinations), which opens the door to model arithmetic: blend two skills, or subtract an unwanted behavior.
• It's the substrate under most open-source fine-tunes you see on the Hugging Face Hub. Understanding LoRA is understanding how ~the whole open-weights ecosystem ships task-specific models.

How it works (the real mechanics)

The core equation

Take any linear layer with a frozen weight matrix W₀ ∈ ℝ^(d×k). A normal fine-tune would learn a full-size update ΔW (also d×k) and use W₀ + ΔW. LoRA's hypothesis is that the update ΔW has a low intrinsic rank — it doesn't need all d·k degrees of freedom. So it factors the update into two thin matrices:

ΔW = B · A
  where  B ∈ ℝ^(d×r),  A ∈ ℝ^(r×k),  and  r ≪ min(d, k)

The adapted forward pass for an input x is then:

h = W₀·x  +  ΔW·x
  = W₀·x  +  (B·A)·x
  = W₀·x  +  (α / r) · B·A·x          ← with the LoRA scaling factor

• `W₀` is frozen — no gradients, no optimizer state. This is where almost all the memory savings come from.
• Only `A` and `B` are trained. Their parameter count is r·(d + k) instead of d·k. For d = k = 4096 and r = 8, that's 8·8192 = 65,536 trainable params versus 16,777,216 — a 256× reduction per matrix.
• `r` is the rank — the single most important LoRA knob. Small r (4–16) is common; bigger r gives more capacity but more params. The whole bet is that small r suffices.
• `α` (`lora_alpha`) is a scaling constant. The effective update is scaled by α/r. Practically, α lets you tune the strength of the adapter without retraining; people often set α = 2r or α = r.

Initialization — the "starts as a no-op" trick

LoRA initializes `A` with a random Gaussian and `B` with zeros (the Microsoft reference init). Therefore at step 0:

ΔW = B·A = 0·A = 0

The adapter contributes nothing at the start, so the model behaves exactly like the pretrained base before any training — a safe, stable starting point. Training then gradually moves B (and A) away from this no-op. (This is why peft's default init_lora_weights=True is "B=0"; setting it False gives a random non-no-op init, only useful for debugging.)

Why low rank is "allowed" — the rank-deficiency hypothesis

The original paper's empirical investigation argues that the weight update needed to adapt a large LM to a new task is intrinsically low-rank — i.e. the task-specific change lives in a small subspace of the full weight space (Hu et al., 2021). They show that even very small r (sometimes r = 1 or 2) can match full fine-tuning on their benchmarks. This is the load-bearing assumption: LoRA works because fine-tuning updates are rank-deficient. (It is an empirical claim, not a theorem — see Caveats.)

Where you put the adapters (target_modules)

You don't have to adapt every layer. The original paper applied LoRA primarily to the attention projection matrices (q_proj, v_proj in particular). In peft you pick target_modules; common choices range from ["q_proj", "v_proj"] (minimal) to all linear layers ("all-linear", including the MLP/FFN projections) for more capacity. More targets = more params = usually better quality, up to a point.

QLoRA — fitting the frozen weights in 4 bits

LoRA already freezes W₀, but W₀ still has to live in GPU memory in 16-bit. For a 65B model that's ~130 GB — too big for one GPU. QLoRA (Dettmers et al., 2023) shrinks the frozen base to 4 bits while keeping the LoRA adapters in higher precision, via three ideas:

1. 4-bit NormalFloat (NF4). Neural-net weights are roughly normally distributed. NF4 is a data type whose 16 quantization levels are placed to be information-theoretically optimal for normally-distributed data (equal probability mass per bin), rather than spaced uniformly. This loses less information than naive 4-bit int quantization. See [[quantization]] for the general theory.
2. Double Quantization. Quantization itself needs metadata — per-block scaling constants. QLoRA quantizes the quantization constants too, shaving another fraction of a bit per parameter on average.
3. Paged Optimizers. Using NVIDIA unified memory, optimizer state is paged between GPU and CPU RAM to absorb the memory spikes (e.g. long sequences) that would otherwise cause an OOM crash.

The training math is then:

h = dequantize(W₀^NF4)·x  +  (α/r)·B·A·x

The forward pass dequantizes the 4-bit base on the fly to compute W₀·x, but gradients flow only into A and B (which stay in 16-bit / bf16). Result: fine-tune a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning quality (Dettmers et al., 2023). The key subtlety: the base is lossy-4-bit, but the trainable adapters are full-precision, so the model can learn to compensate for some quantization error during training.

Pseudocode in prose (one LoRA linear layer)

Store the original weight W₀ (frozen, possibly 4-bit). Allocate A as a random r×k matrix and B as a zeros d×r matrix; mark only A,B trainable. On forward: compute the base output W₀·x (dequantizing W₀ first if it's quantized), compute the low-rank path B·(A·x) — note you do A·x first so you never materialize the d×k product — scale it by α/r, and add the two. On backward: gradients update only A,B. When done: optionally fold the adapter into the base, W' = W₀ + (α/r)·B·A, and ship a single merged matrix.

Key ideas & tradeoffs (the PEFT family)

LoRA is one point in a larger design space. The peft library groups them:

Notable LoRA-family refinements (all in peft):

• rsLoRA (Rank-Stabilized LoRA). The original α/r scaling causes forward/backward magnitudes to shrink as you raise r, which can stall learning at high rank. rsLoRA proves the stable scaling is `α/√r` (Kalajdzievski, 2023, [arXiv:2312.03732]). In peft: use_rslora=True. Use it when you want to crank rank up.
• DoRA (Weight-Decomposed LoRA). Decomposes each weight into a magnitude scalar m and a unit-norm direction V/‖V‖, then applies LoRA to the direction only and learns the magnitude separately: W = m · (V / ‖V‖). This better mimics how full fine-tuning moves magnitude and direction independently, improving accuracy — especially at low rank — with no extra inference cost once merged (Liu et al., 2024, [arXiv:2402.09353]; ICML 2024 Oral). In peft: use_dora=True. Costs more during training than plain LoRA.
• Smarter initializations. peft ships PiSSA, OLoRA, LoftQ, EVA, CorDA, LoRA-GA — variants that initialize A,B from an SVD of the base weights or of gradients instead of randomly, to converge faster or reduce quantization error. LoRA-GA reports 2–4× faster convergence by aligning the init with the full-fine-tune gradient direction ([arXiv:2407.05000]). These are optimizations on the starting point, not the core mechanism.

Choosing the right knobs (rules of thumb):

• `r` (rank): start at 8–16. Raise it if the task is hard / data-rich; if you go high, switch on rsLoRA.
• `α`: a common default is α = 2·r. It's a strength dial; you can tweak it post-hoc.
• `target_modules`: attention-only (q,v) is the cheapest; add the MLP projections or use all-linear for harder tasks.
• `lora_dropout`: small (0.05–0.1) regularizes; helps on small datasets.

When to use LoRA/PEFT vs. full fine-tuning

Reach for LoRA/QLoRA when:

• You're GPU- or budget-constrained (this is most people).
• You need many specializations off one base (per-customer / per-persona).
• You want a small, shippable artifact and the option to merge with zero inference cost.
• The task is "specialize / adapt style / add a skill" rather than "teach a fundamentally new capability."

Consider full fine-tuning when:

• You're injecting a large amount of new knowledge or a genuinely new capability (low-rank corrections may not have the capacity).
• You have the compute and the task is your core product where every last quality point matters.
• Research suggests LoRA and full fine-tuning learn structurally different solutions even when benchmark scores match (see Caveats) — so for some high-stakes cases the equivalence is not guaranteed.

Honest caveats & open questions

• "On par with full fine-tuning" is benchmark-dependent, not universal. The original results were strong on the tasks tested, but LoRA has a capacity ceiling: the update is constrained to rank r. For tasks needing broad new knowledge, low rank can underfit, and you'll see a gap. Don't treat "LoRA = full FT" as a law.
• Equivalence may be an illusion. "LoRA vs Full Fine-tuning: An Illusion of Equivalence" ([arXiv:2410.21228]) shows LoRA can introduce "intruder dimensions" — singular directions absent from full fine-tuning — that hurt out-of-distribution generalization and worsen forgetting, even when in-task accuracy matches. peft even ships a post-hoc intruder-dimension mitigation utility. So matching accuracy ≠ matching the underlying solution.
• The rank-deficiency hypothesis is empirical. "Fine-tuning updates are low-rank" is observed on certain models/tasks, not proven in general. New domains may violate it.
• QLoRA's 4-bit base is lossy. NF4 + double-quant minimize the loss, and the adapters compensate during training, but you are still fine-tuning on top of a degraded base. For the highest-fidelity work, 16-bit LoRA (or full FT) can edge it out. LoftQ exists precisely to reduce this quantization-induced gap.
• Hyperparameter sensitivity. Quality depends on r, α, target_modules, learning rate, and init scheme. There's no universal recipe; expect to sweep.
• Adapter merging isn't free of artifacts. Combining many adapters (ties, dare, cat) can collide or blow up rank; merged behavior can surprise you.
• Active research front. Init schemes (PiSSA/EVA/CorDA), scaling (rsLoRA), decomposition (DoRA), and serving (activated-LoRA / multi-adapter inference on vLLM) are all moving fast. The "best" PEFT setup in mid-2026 is a moving target — treat specific defaults as provisional.

How it connects to OpenAlice

This is grounded but speculative where noted — OpenAlice's specialization story today is prompt-level, not weight-level, and that's a deliberate honest line to draw.

• The adapter-per-persona pattern mirrors OpenAlice's architecture. Alice already runs one base model with per-chat mode axes — InteractionMode (Companion vs Worker), PersonalityMode (6 moods), ChatMode — layered as prompt/overlay state, not separate model weights (see reference-alice-three-mode-axes-2026-06-08). LoRA is the weight-level analog of exactly this idea: a frozen base + swappable, cheap, per-context adapters. If Alice ever needs behavior that prompting can't reliably hold, a LoRA-per-persona library is the natural next rung — same "one base, many thin specializations" shape Alice already uses, moved from the prompt into the weights.
• It fits the cost-ladder mindset. OpenAlice's [[model-routing]] and cost-ladder work is about not paying frontier price for every request. PEFT is the training-time sibling: not paying full-fine-tune cost for every specialization. Both are "match the task to the cheapest sufficient resource."
• Provider reality check. Alice's default serving path is Codex/OpenAI OAuth (gpt-5.x), where you can't drop in your own LoRA weights — fine-tuning there is the provider's hosted offering, not local LoRA. So today LoRA is most relevant to OpenAlice as (a) conceptual literacy for the Academy, and (b) the enabling tech if/when the lab fine-tunes an open-weights model locally on the no-GPU-but-62 GB-RAM blal.de box — where QLoRA's single-GPU / low-memory story is the only thing that would make local fine-tuning feasible at all.
• Academy value. For learners, LoRA is the cleanest on-ramp to "how do people actually customize LLMs cheaply?" — pairs naturally with [[quantization]], [[attention-and-transformers]], and [[llm-from-scratch]].

See also

• [[quantization]] — the 4-bit machinery (NF4, double-quant) under QLoRA.
• [[attention-and-transformers]] — the q/k/v projections LoRA usually targets.
• [[llm-from-scratch]] / [[microgpt-build-an-llm-from-scratch]] — how the base model you're adapting was trained.
• [[model-routing]] — the inference-time cost-optimization sibling to PEFT's training-time cost optimization.
• [[mixture-of-experts]] — another "don't activate the whole model" idea, but at inference rather than fine-tuning time.

Sources

1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021 — arXiv:2106.09685↗
2. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, 2023 — arXiv:2305.14314↗
3. Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, 2024 (ICML Oral) — arXiv:2402.09353↗
4. Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA), 2023 — arXiv:2312.03732↗
5. HuggingFace PEFT — adapters conceptual guide & LoraConfig reference — huggingface.co/docs/peft↗
6. (referenced) LoRA vs Full Fine-tuning: An Illusion of Equivalence — arXiv:2410.21228↗; LoRA-GA — arXiv:2407.05000↗