LoRA
also called Low-Rank Adaptation, PEFT
Cheap fine-tuning that trains a small add-on instead of the whole model.
A fine-tuning method that freezes the original weights and trains a small pair of low-rank matrices alongside them. You get most of the benefit of fine-tuning at a fraction of the cost and memory.
Because the base model is untouched, you can keep many adapters and swap them per request — one model, several personalities or specialities.
The reason LoRA became the default is not only cost, it is operations. Adapters are small files you can version, ship and swap at request time, so one base model can serve several behaviours. That is a very different deployment story from maintaining several full copies of a model.
A per-tenant adapter
Fifty customers each want the assistant to follow their house style.
Train fifty small adapters against one base model rather than fifty models. Storage drops from terabytes to megabytes, and onboarding a new customer becomes a training job measured in minutes rather than a deployment.
- paperLoRA: low-rank adaptation of large language models ↗
Freeze the pretrained weights, train small low-rank matrices beside them. The origin of parameter-efficient fine-tuning, and why adapters are cheap to ship and swap.
- paperQLoRA: efficient finetuning of quantized LLMs ↗
Backpropagate through a frozen 4-bit quantised base model into LoRA adapters, which is what makes single-GPU fine-tuning of large models ordinary.