Vol. XVII · Supervised Fine-Tuning Plate 7 of 10QLoRA, 4-bit quantisation, NormalFloat, NF4, double quantisation, paged optimisers, memory arithmeticCE

Fine-Tuning a Quantised Model.

Freeze the base in four bits, train the adapters in sixteen: a 65-billion-parameter fine-tune on one GPU.

LoRA leaves the frozen base as the dominant memory cost. Dettmers and colleagues’ QLoRA (2023) quantised it: the base weights are stored in a 4-bit format and dequantised to BF16 on the fly in each layer’s forward and backward pass, while gradients flow through them into BF16 LoRA adapters. Three ideas made it work. NormalFloat-4 places its 16 levels at the quantiles of a normal distribution, which matches how weights are distributed. Each block of 64 weights has its own scale, and double quantisation stores those scales themselves in 8 bits, saving about 0.37 bits per parameter. Paged optimisers move optimiser state to host memory during memory spikes.

The result was a 65-billion-parameter model fine-tuned on a single 48 GB GPU. The workshop does the memory arithmetic for a chosen model, base format and adapter rank, and compares it with the memory of common accelerators.

Below: memory for fine-tuning the chosen model, split into base weights, quantisation scales, adapters with their optimiser state, and checkpointed activations, for each base format, to compare with GPUs of 24, 48 and 80 GB.

What you will learn

Explain QLoRA’s division into a 4-bit base and BF16 adapters.
Compute the memory of NF4 with and without double quantisation.
Total the memory of a QLoRA fine-tune.
Compare it with full fine-tuning and BF16 LoRA.

Workshop

Base weights: BF16 (16 bits), INT8 with one FP32 scale per 64 (8.5), NF4 with one FP32 scale per 64 (4.5), NF4 with double quantisation (4.127). Adapters of the chosen rank on all seven matrices of every layer, 16 bytes per adapter parameter. Activations with every layer checkpointed: 2 bytes × d per token per layer, for 2 048 tokens. Full fine-tuning shown for comparison at 16 bytes per parameter.

Four bits frozen, sixteen trained

For the 65-billion-parameter LLaMA of the QLoRA paper with rank-64 adapters, a 4-bit NormalFloat base with double quantisation brings the total to 49.2 GB in this accounting — at the edge of a 48 GB GPU, which is where the paper’s paged optimisers, moving optimiser state to host memory during spikes, earned their place; double quantisation alone saves 3.0 GB. The same LoRA on a BF16 base needs 146.0 GB, and full fine-tuning needs 1.0 TB for its states before any activation — dozens of GPUs. For Llama 3 8B, QLoRA needs 7.4 GB, within a consumer GPU; for Llama 3 70B, 52.3 GB.

The price is not memory but compute and fidelity. Every forward and backward pass dequantises each block of weights before multiplying, so a QLoRA step is slower than a LoRA step on a BF16 base; and the base the adapters learn against is a slightly perturbed copy of the original. The paper found 4-bit NF4 with adapters on every linear layer recovered the quality of 16-bit LoRA on its benchmarks, and the method made fine-tuning of large open models something an individual could do.

The adapters are trained against the dequantised weights, so they partly learn to compensate for the quantisation error — one reason a QLoRA adapter is best served on the same 4-bit base it was trained with.

Terms

QLoRA: LoRA adapters trained through a frozen, 4-bit-quantised base model.
NormalFloat-4: A 4-bit format with levels at the quantiles of a normal distribution.
Double quantisation: Quantising the per-block scales themselves to save memory.
Paged optimiser: Optimiser state that spills to host memory during memory spikes.

QLoRA keeps the frozen base in about 4.1 bits per weight and trains BF16 adapters through it, putting a 65B fine-tune on one 48 GB GPU.

Sources

Dettmers, Pagnoni, Holtzman and Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs”, NeurIPS 2023.
Touvron et al., “LLaMA”, arXiv:2302.13971, 2023 (65B configuration).