Instruction Tuning
also called SFT, Supervised fine-tuning
The stage that turns a text predictor into something that follows requests.
Fine-tuning on pairs of instructions and ideal responses so the model learns to answer requests rather than merely continue text. This is what makes a base model usable as an assistant.
A base pretrained model will happily continue your question as if it were a document. Instruction tuning teaches the frame "you are being asked, now respond".
This is the stage that converts a text predictor into something you can give an instruction to, and the evidence is unusually clean: training on many tasks phrased as instructions improved performance on tasks the model had never been trained on. Generality came from the format of the training data, not its breadth alone.
Why a base model feels broken
Someone tries an un-tuned base model from a model hub and finds it ignores instructions.
It is not a worse model — it is the wrong stage. A base model continues text. Ask it a question and it produces more questions, because that is what the training distribution contained. Instruction tuning is the missing step.
- paperFinetuned language models are zero-shot learners ↗
Instruction tuning: training on many tasks phrased as instructions improved results on tasks the model had not been trained on.
- paperTraining language models to follow instructions with human feedback ↗
InstructGPT: supervised fine-tuning on demonstrations, then a preference model, then RLHF. Also the result that raters preferred a much smaller tuned model to raw GPT-3.
Fine-tuning
Extra training on your own examples to change how the model behaves.
RLHF · Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
System Prompt · System message
The standing instructions the user does not see.
Instruction Following
The model’s ability to actually obey what you asked.