Fine-tuning
Extra training on your own examples to change how the model behaves.
Continuing training on a smaller, task-specific dataset so the model picks up a style, format or domain speciality. It changes weights, so it changes the model itself, not just the prompt.
Fine-tuning is good at teaching form — output shape, tone, a fixed taxonomy. It is a poor way to teach new facts; for that, retrieval is cheaper, faster to update, and auditable.
The decision rule that saves the most wasted effort: fine-tuning teaches form, retrieval supplies facts, and prompts carry instructions. If the problem is "it does not know our internal terminology", fine-tuning is the wrong tool and an updated retrieval corpus is the right one.
Format, not knowledge
Support replies are correct but written in a tone that annoy customers.
A few hundred examples of ideal replies teaches the register reliably. The same fine-tune would not have taught the model a single new product fact. Splitting the two problems is what makes each of them cheap to solve.
When to walk it back
A narrow fine-tune made the model much better at one task and noticeably worse at general coding.
That is catastrophic forgetting, and it is the argument for adapters rather than full fine-tuning: keep the base weights untouched and swap the adapter per task, so improving one job cannot degrade another.
Fine-tuning to inject facts that change weekly. You will be retraining forever.
- paperTraining language models to follow instructions with human feedback ↗
InstructGPT: supervised fine-tuning on demonstrations, then a preference model, then RLHF. Also the result that raters preferred a much smaller tuned model to raw GPT-3.
- paperLoRA: low-rank adaptation of large language models ↗
Freeze the pretrained weights, train small low-rank matrices beside them. The origin of parameter-efficient fine-tuning, and why adapters are cheap to ship and swap.
- paperTraining a helpful and harmless assistant with reinforcement learning from human feedback ↗
The foundational helpful-and-harmless RLHF recipe, and the source of the alignment-tax discussion: how much capability you lose by training for harmlessness.
Training
The expensive one-off process that sets the model’s parameters.
Instruction Tuning · SFT
The stage that turns a text predictor into something that follows requests.
LoRA · Low-Rank Adaptation
Cheap fine-tuning that trains a small add-on instead of the whole model.
RAG · Retrieval-Augmented Generation
Look things up, then put what you found in the prompt.
Distillation
Training a small model to imitate a big one.