Vol. XVII · Supervised Fine-Tuning Plate 6 of 10LoRA, low-rank adaptation, parameter-efficient fine-tuning, rank, adapters, optimiser memory, mergingCE

LoRA.

Freeze the model; learn a thin correction beside each matrix. A billion-parameter update becomes a few million.

Full fine-tuning updates every weight and so needs the full 16 bytes per parameter of training (plate XIV.1). Hu and colleagues’ LoRA (2021) observed that the change fine-tuning makes to a weight matrix tends to have low intrinsic rank, and learned it directly as a product: W′ = W + (α/r)·BA, with A of shape d_in × r and B of shape r × d_out, B initialised to zero so training starts from the base model exactly. Only A and B are trained; W stays frozen in whatever precision it is stored.

On GPT-3 175B, LoRA cut the trainable parameters by a factor of 10 000 and the training memory by 3 against full fine-tuning with Adam. Afterwards BA can be added into W, so the served model is no larger or slower; or many adapters can be kept beside one base and swapped per request. The workshop counts the parameters and memory for adapters of a chosen rank on a chosen set of matrices.

Below: left, trainable parameters and training memory for full fine-tuning and for LoRA at the chosen rank. Right, how the trainable count grows with rank.

What you will learn

Write the LoRA update.
Count trainable parameters for a rank and a set of matrices.
Compare training memory with full fine-tuning.
Explain merging and swapping adapters.

Workshop

Adapters on every layer’s attention projections (q, k, v, o) or on those and the three MLP matrices. Memory: full fine-tuning at 16 bytes per parameter; LoRA with the frozen base in BF16 (2 bytes per parameter) plus 16 bytes per adapter parameter. Activations are left out of both. Adapter file: BF16.

A thin correction beside each matrix

Rank-16 adapters on all seven matrices of every Llama 3 8B layer add 41.9 million trainable parameters, 0.52% of the model. Training memory without activations falls from 128.5 GB to 16.7 GB, 7.7 times less, almost all of it now the frozen BF16 weights; the adapter file is 83.9 MB. On attention only, the count falls to 13.6 million. For the 70-billion-parameter model the same rank adds 207 million and takes training from 1.1 TB to 144.4 GB.

Why should a low rank suffice? The paper’s evidence is empirical: the updates full fine-tuning makes have most of their energy in a few directions, and LoRA at small rank matches it on many tasks. It does not always — on tasks that need new knowledge rather than a new style, or on long continued training, full fine-tuning tends to learn more, and LoRA tends to forget less (plate XVII.8). The zero initialisation of B matters too: at step zero the model is exactly the base, so fine-tuning starts where pre-training left off.

Because the adapters are separate, one served base model can carry hundreds of them — one per customer or task — loaded per request; the base weights, which dominate memory, are shared.

Terms

LoRA: Low-rank adaptation: learning ΔW = BA beside frozen weights.
Rank: The inner dimension r of the adapter product.
Adapter: The small set of trained parameters added to a frozen model.
Merging: Adding BA into W so the adapted model runs at the base model’s cost.

LoRA trains r(d_in + d_out) parameters per matrix beside frozen weights: a fraction of a per cent of the model, with memory to match.

Sources

Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, ICLR 2022.
Llama 3 configurations, arXiv:2407.21783, Table 3.