LoRA & Quantization Visualization | Low-Rank Adapters, int8, int4

The idea: train fewer numbers, store fewer bits

A large language model is mostly weight matrices: a 7-billion-parameter model has 7·10⁹ numbers. Two tricks make such a model affordable. LoRA (low-rank adaptation) fine-tunes it by training a small add-on next to each frozen matrix instead of the matrix itself. Quantization stores each weight in 8 or 4 bits instead of 16 or 32, so the model takes a half or a quarter of the memory. The canvas shows both on matrices small enough to read; pick one with the show dropdown.

The input x goes two ways: through the frozen d by d matrix W, and through the trainable path A (r by d, squeezing x to r numbers) then B (d by r), scaled by alpha over r. The two results are added to give y. For d 4096 and r 8 the adapter has 65,536 numbers instead of 16.8 million. — LoRA: the frozen W keeps its path; a thin trainable side path B·A is added to its output.

LoRA: a low-rank update next to a frozen W

Full fine-tuning changes every number of a weight matrix W (d × d). LoRA keeps W frozen and learns the change as a product of two thin matrices: ΔW = (α/r)·B·A, with B d × r and A r × d. The rank r is small (4 to 64). The layer computes

y = W·x + (α/r)·B·(A·x)

A squeezes the d inputs into r numbers and B spreads them back to d outputs, so ΔW has rank at most r. α (lora_alpha) is a fixed constant; dividing it by r keeps the size of the update similar when you change r. The canvas uses d = 8, α = 4 and r = 1, 2 or 4.

One layer	Full fine-tune	LoRA
canvas: d = 8, r = 2	64	8·2 + 2·8 = 32
d = 4096, r = 8	16,777,216	65,536 (0.39%)
7B model, r = 8 on W_Q and W_V in 32 layers	~7,000,000,000	4,194,304 (0.06%)

The saving is larger than the parameter count suggests: the optimizer (Adam) keeps two extra numbers per trained weight, plus its gradient. A full fine-tune of a 7B model needs about 16 bytes per weight (over 100 GB); with LoRA those extras exist only for the adapter.

Start at zero, then train

A starts random and B starts at zero, so at step 0 B·A = 0 and the adapted model is exactly the base model. Training then starts from what the model already knows. (Demo 1: LoRA at init: y equals W·x.)

The toy task on the canvas: the layer should behave like W + T, where T is a hidden matrix of rank 2. The loss is the mean squared error of the outputs on 16 fixed inputs, and gradient descent (learning rate 0.5) updates only B and A. The gradients are ∂L/∂B = (α/r)·G·Aᵀ and ∂L/∂A = (α/r)·Bᵀ·G, with G the gradient of the loss with respect to ΔW; W gets none. Because B = 0 at the start, A gets no gradient in the first step: B moves first.

rank r	trainable	loss at start	after 100 steps
1	16	0.424	0.094 (stuck)
2	32	0.424	0.003
4	64	0.424	0.007

A rank-1 adapter can only make rank-1 changes, so it cannot fit a rank-2 T and the loss stalls (Demo 3: rank 1 cannot fit a rank-2 change). The LoRA paper's finding is that the change fine-tuning needs is usually of low rank, so a small r is often enough.

Merge after training

When training is done, W' = W + (α/r)·B·A is computed once and replaces W. The merged layer gives the same outputs with one matrix multiply, so inference costs exactly what the base model costs (Demo 2: train rank 2, merge, same output). Keeping B and A separate instead lets one base model in memory serve many small adapters, one per task or customer, swapped per request.

Quantization: scale and zero-point

Quantization replaces each float weight by a small integer code. With affine (asymmetric) quantization to b bits, a group of weights with range [min, max] gets a scale s and a zero-point z:

s = (max − min) / (2^b − 1), the size of one step
z = round(q_min − min/s), the code that stands for 0.0
q = clamp(round(w/s) + z, q_min, q_max): int8 codes run from −128 to 127, int4 from −8 to 7
ŵ = s·(q − z): the dequantized value used in the computation

A number line from min minus 0.488 to max 0.352 with the 16 int4 levels, codes minus 8 to 7, spaced by the step s equal to 0.056. Zero falls exactly on code 1, the zero-point. A weight 0.204 rounds to the nearest level 0.224, an error of 0.020, at most s over 2. — int4 on the canvas matrix: 16 levels 0.056 apart; every weight snaps to the nearest one, so the error is at most s/2.

The range always includes 0, so a weight of exactly 0 stays exact. Rounding loses at most half a step, s/2, per weight. On the canvas matrix, per-tensor int8 has s = 0.0033 and a mean error of 0.5% of a typical weight; int4 has only 16 levels, s = 0.056, and the mean error grows to 8.4% (Demo 4: int8 vs int4). The error matrix is tinted from white (no error) to red.

Symmetric quantization is the special case z = 0 with s = max|w| / 127; it is simpler to compute with, and is common for weights.

Outliers and scaling granularity

LLM weight matrices (and even more their activations) contain a few outliers, values many times larger than the rest. One outlier sets the max of its group and stretches the step for every weight in the group. With one weight of 2.5 in the canvas matrix and per-tensor int4, s jumps from 0.056 to 0.199: the mean error of the normal weights rises to 26%, and 11 of them round to 0 (Demo 5: one outlier ruins per-tensor int4).

Three copies of a 4 by 8 matrix with one outlier. Per-tensor: one scale for all 32 weights, all of them pay, error 26 percent. Per-channel: one scale per row, only the outlier's row pays, 10.6 percent. Per-group of 4: only the outlier's group of 4 pays, 6.7 percent. — The smaller the group that shares a scale, the fewer weights an outlier can hurt.

The fix is to give smaller groups their own scale and zero-point:

int4, outlier 2.5	(s, z) pairs	mean error	weights → 0
per-tensor	1	26.2%	11
per-channel (row)	4	10.6%	5
per-group (4 weights)	8	6.7%	3
no outlier, per-group	8	4.4%	3

(Demo 6: per-channel and per-group scaling.) Real 4-bit formats use groups of 64 or 128 weights; their fp16 scales add about 0.25 to 0.5 bit per weight. LLM.int8() keeps outlier activation columns in fp16 instead; GPTQ and AWQ choose the rounding so that the layer's output, not each weight, stays close.

What it saves

7B model weights	bits per weight	size
fp32	32	28 GB
fp16 / bf16	16	14 GB
int8	8	7 GB
int4	4	3.5 GB
int4 + fp16 scale and zero per 64 weights	4.5	3.9 GB

Generating text one token at a time reads every weight once per token, so it is limited by memory bandwidth: fewer bytes per weight also means faster tokens, as long as the dequantization is fused into the matrix-multiply kernel. See Floating Point for why fp16 and bf16 already differ in range and precision.

QLoRA: both at once

QLoRA stores the frozen base model in 4 bits (the NF4 format, whose 16 levels are placed for normally distributed weights, with double-quantized scales) and trains LoRA adapters in bf16 on top. Each frozen weight is dequantized on the fly for the forward and backward pass; only B and A get gradients. That fits the fine-tuning of a 65B model on a single 48 GB GPU.

See also How a Large Language Model Works for where these matrices sit, Self-Attention for W_Q and W_V, the matrices LoRA is most often put on, Matrix Multiplication for B·A, SVD and PCA for low-rank approximation, and Neural Network Forward Pass for W·x.

See also Mixture of Experts: an MoE needs the memory of all its experts but the compute of only k, one reason quantization matters for models like Mixtral.

LoRA adapters are a common way to run RLHF or DPO cheaply, with the base weights as the reference model; see RLHF vs DPO.

What the page leaves out

Real LoRA training uses Adam, mini-batches, dropout on the adapter and a real loss (next-token cross-entropy), and A is initialised with Kaiming-uniform in the PEFT library. Variants such as DoRA, LoRA+ and rsLoRA (scale α/√r) are not shown. On the quantization side the page leaves out activation quantization (W8A8), calibration data, GPTQ's error compensation, AWQ's per-channel rescaling, NF4 and FP8 formats, quantization-aware training and the integer matrix-multiply kernels that do the work on a GPU.