LoRA and parameter-efficient tuning

Two ways into the same idea

Before You Start

The Core Idea

LoRA freezes the original weights and trains two small matrices beside them. In this lesson you see how many parameters it saves, why a small rank is enough, how it starts at zero and merges, and how to set it up with a library.

1Freeze W, learn A and B

Instead of changing all of W, we learn an update ΔW = A × B. A is tall and thin, B is short and wide, and their shared width is the rank r, usually 8 or 16.

Look at the picture: a 4096 by 4096 matrix has 16.8 million parameters, and at rank 8 we train only 0.07 million. The second bar is barely visible.

d = 4096                       # one projection matrix in a 7B model is 4096 x 4096
full = d * d
for r in [4, 8, 16, 64]:
    lora = d * r + r * d       # A is d x r, B is r x d
    print(f"rank {r:2}: {lora:>9,} trainable instead of {full:,}  ({lora / full:.2%})")

One 4096 by 4096 matrix has 16.7 million parameters. At rank 8 we train only 65 thousand, under 0.4%. Across a whole model that means tens of millions instead of billions, so one mid-range card's memory is enough, and the output is a small file of a few tens of megabytes instead of a full copy of the model.

2Why a small rank is enough

A pretrained model already knows a lot, and tuning does not teach it from scratch. It tilts it in a few directions: this format, this style, these categories. The rank is the number of those directions.

Look at the picture: the error falls quickly up to rank 6, then nearly stops at 0.20. Beyond it there is only noise.

A "real" update built from 6 directions plus light noise. We approximate it at different ranks and measure the error.

import numpy as np

rng = np.random.default_rng(0)
d = 256
true_rank = 6
# a "real" fine-tuning update: mostly a few directions, plus a little noise
update = rng.normal(size=(d, true_rank)) @ rng.normal(size=(true_rank, d)) / d
update += rng.normal(scale=0.002, size=(d, d))

u, s, vt = np.linalg.svd(update)
for r in [1, 2, 4, 6, 8, 16]:
    approx = (u[:, :r] * s[:r]) @ vt[:r]
    err = np.linalg.norm(update - approx) / np.linalg.norm(update)
    print(f"rank {r:2}: relative error {err:.3f}   parameters {2 * d * r:>6,} of {d * d:,}")

The error falls quickly up to rank 6, then nearly stops at about 0.20. What remains is the noise, and higher ranks only learn that. In real tuning you do not know the "true" rank, so you try 8, 16 and 32 on the validation set and pick where improvement stops.

3Starting at zero, then merging

The output becomes x W + (α / r) x A B. We start A small and random and B at zeros, so at the first step the model is exactly the original.

Look at the picture: training changes only A and B, then we merge W + (α / r) A B into one matrix for deployment.

import numpy as np

rng = np.random.default_rng(1)
d, r, alpha = 64, 8, 16
W = rng.normal(size=(d, d)) / np.sqrt(d)      # frozen pretrained weight
A = rng.normal(size=(d, r)) * 0.01            # trainable, small random
B = np.zeros((r, d))                          # trainable, zeros: the update starts at nothing
scale = alpha / r
x = rng.normal(size=(4, d))

def lora_forward(x):
    return x @ W + scale * (x @ A) @ B

print("before training, LoRA output equals the base model:", bool(np.allclose(lora_forward(x), x @ W)))

B = rng.normal(size=(r, d)) * 0.05            # pretend training changed B
merged = W + scale * A @ B                    # fold the adapter into the weight
print("after merging, one matrix gives the same output:", bool(np.allclose(lora_forward(x), x @ merged)))
print("extra cost at inference after merging: 0 extra matrices")

B starts as zeros, so at the first training step the model is exactly the original, and training does not begin by wrecking what it knows. After training, merging A B into W gives one matrix with the same outputs, so LoRA costs nothing at inference.

4Rank, scale and layers

The rank r is the matrices' width, and α sets the update's strength. A common start is r = 16 and α = 32. You put adapters on at least the attention projections q, k, v and o, and adding the feed-forward usually helps at the cost of more parameters.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", torch_dtype="bfloat16")
config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()   # about 1% of the model is trainable

# after training:
# merged = model.merge_and_unload(); merged.save_pretrained("tuned-merged")

target_modules picks the layers, and their names differ between model families, so print model and look. And merge_and_unload merges the adapter into the model to deploy it as an ordinary model.

How It Works, Step by Step: how LoRA trains one layer

Going deeper

QLoRA goes a step further: it loads the frozen base model at 4-bit precision instead of 16, cutting its memory to a quarter, and trains LoRA adapters at normal precision on top. That made tuning a seven-billion-parameter model possible on a single 16-gigabyte card, with a small quality loss that you measure.

Worked example

The idea: instead of changing the whole weight matrix W, we learn an update ΔW = A × B, where A is tall and thin and B short and wide. Their shared width is the "rank" r, and it is small: usually 8 or 16.

Try it without code

Myth vs Reality

Try it yourself

Quick review cards

Check Your Understanding