Vol. XVII · Supervised Fine-Tuning Plate 8 of 10catastrophic forgetting, fine-tuning cost, learning rate, LoRA, format overfitting, alignment taxLE

What Fine-Tuning Costs.

Every step towards the new format is a step away from the old one. How far to go is a choice.

A network trained on one task and then another tends to lose the first: catastrophic forgetting. Fine-tuning a language model is a mild version of the same thing. The new data is narrow — one format, a few thousand examples — and nothing in the loss asks the model to keep doing what it did before, so abilities that the fine-tuning data does not exercise can degrade. The InstructGPT paper called the resulting drop on standard benchmarks an alignment tax, and reduced it by mixing pre-training data into the fine-tuning.

The workshop measures it in the post-training world. Before fine-tuning the base completes every worksheet sum; the workshop fine-tunes it to chat three ways and tracks, step by step, its held-out chat accuracy and its loss and accuracy on worksheets it no longer trains on.

Below: left, held-out chat accuracy; right, loss on worksheet text, as fine-tuning proceeds by the chosen method. Readouts compare worksheet accuracy before and after.

What you will learn
Workshop
Live fine-tuning from the recorded base on the 30 human-written chat examples, loss on answers only. Worksheet loss: mean next-token loss on fixed worksheet text. Worksheet accuracy: the share of the 100 sums completed correctly after “1+1=2. a+b=”. Chat accuracy on the 30 held-out pairs.

The cost of moving the weights

Fine-tuning all weights at 0.002 for 150 steps reaches 53% on held-out chat questions, while worksheet accuracy falls from 100% to 33% and worksheet loss rises from 0.69 to 4.49: the model now expects “|” where a worksheet has “. ”, and some sums are lost outright. At a quarter of the learning rate it forgets less (71% of worksheet sums kept, loss 1.82) and learns less (17%). A rank-4 LoRA, training 4,608 parameters, keeps 51% of the worksheet sums at 47% chat accuracy.

The trade is general: how much a fine-tune forgets tracks how far it moves the weights, and a study of LoRA against full fine-tuning on real models found the same — LoRA learns less of a new domain and forgets less of the old. The remedies are the obvious ones: smaller learning rates and fewer steps, parameter-efficient methods, mixing some pre-training or general data into the fine-tuning set, and evaluating the old abilities, not just the new one. Fine-tuning has subtler costs as well: a model can learn the confident style of its examples without their substance, or over-fit their format so that it answers every question in the same shape.

Forgetting here is partly format: the worksheet loss rises because the model now ends a sum with the chat format’s “|”. Some of what fine-tuning “forgets” is reachable again with the right prompt — the same distinction between a capability and its surface form as in plate XVII.1.

Terms

Catastrophic forgetting
Losing an earlier skill when training on a new task.
Alignment tax
The drop in general capability that alignment training can cause.
Replay
Mixing earlier training data into new training to preserve old skills.
Format over-fitting
Learning the shape of the examples so rigidly that every answer takes it.
Fine-tuning forgets in proportion to how far it moves the weights; smaller steps, low-rank adapters and replayed data trade some learning for less forgetting.

Sources

  1. McCloskey and Cohen, “Catastrophic Interference in Connectionist Networks”, 1989.
  2. Ouyang et al., NeurIPS 2022, section 3.6 (alignment tax; mixing in pre-training gradients).
  3. Biderman et al., “LoRA Learns Less and Forgets Less”, TMLR 2024.