Catastrophic Forgetting
Fine-tuning that improves one thing and breaks others.
When training on a narrow task erases general capability the model previously had. The model gets better at your dataset and worse at everything adjacent.
This is why fine-tuning calls for held-out general evaluations, not just task accuracy. LoRA reduces the risk by leaving the base weights untouched, but does not eliminate it.
This is the concrete argument for adapters over full fine-tuning, and for evaluating a fine-tune on more than the task it targets. Train on your task, then run the general capability evals you cared about before; the regression is real and it is easy to miss because you were not looking for it.
The tune that broke the rest of the product
Fine-tuning for a billing-specific tone degrades general instruction following.
Run the pre-existing evals after any fine-tune, not just the new task’s evals. If general capability dropped, either reduce the training scope, use a low-rank adapter so the base weights stay intact, or accept the trade explicitly.
- paperLoRA: low-rank adaptation of large language models ↗
Freeze the pretrained weights, train small low-rank matrices beside them. The origin of parameter-efficient fine-tuning, and why adapters are cheap to ship and swap.
- paperTraining language models to follow instructions with human feedback ↗
InstructGPT: supervised fine-tuning on demonstrations, then a preference model, then RLHF. Also the result that raters preferred a much smaller tuned model to raw GPT-3.
Fine-tuning
Extra training on your own examples to change how the model behaves.
LoRA · Low-Rank Adaptation
Cheap fine-tuning that trains a small add-on instead of the whole model.
Overfitting
Memorising the training data instead of learning the pattern.
Evaluation · Evals
How you know the system is actually working.