EXPLAINER

LoRA and Parameter-Efficient Fine-Tuning (PEFT)

techniquefreshness: evolvingsources not yet verified

LoRA and other PEFT methods adapt a large model by training a small number of added parameters while the original weights stay frozen.

Short answer: LoRA adds small low-rank adapter matrices to a frozen model and trains only those, cutting memory and storage dramatically. QLoRA combines it with a quantised base. Adapters can be swapped per task.

What is it?

Fully fine-tuning a giant model means updating billions of numbers. LoRA freezes the original and trains a tiny add-on instead — often a small fraction of the size — so tuning fits on far smaller hardware and you can keep many small add-ons for different tasks.

Also known as: LoRA, QLoRA, PEFT, low rank adaptation, adapter fine-tuning, parameter efficient fine tuning, DoRA, prompt tuning.

Technical definition

LoRA (Hu et al., 2021) reparameterises weight updates as ΔW = BA with low rank r, injected into attention (and often MLP) projections; at inference adapters can be merged into the base weights or served separately. QLoRA (Dettmers et al., 2023) fine-tunes adapters on a 4-bit quantised base. Other PEFT families: adapters (Houlsby et al., 2019), prefix/prompt tuning, DoRA. Quality is often close to full fine-tuning on many tasks but not always, particularly for large distribution shifts.

Entity type: technique · Freshness class: evolving.

When should I use it?

Default for adapting open models on modest hardware.

Implementation patterns

  • Target attention (and optionally MLP) projections with rank 8–64 as a starting range to test.
  • QLoRA when memory is tight.
  • Merge adapters for deployment or serve multiple adapters on one base.
  • Keep an evaluation set per adapter.

Technologies and frameworks

Commonly associated: Hugging Face PEFT, TRL, bitsandbytes, Axolotl, Unsloth. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Choose a base model and licence.
  2. Prepare chat-template-formatted data.
  3. Start with a modest rank and standard LR.
  4. Evaluate vs the base and a prompt-only baseline.
  5. Decide merge vs separate serving.

Common failure modes

  • Rank too low to learn the task, or alpha/LR mis-set.
  • Merging into a quantised base without care, degrading quality.
  • Adapter/base version mismatch.
  • Overfitting small datasets.

Troubleshooting

  • Adapter barely changes behaviour — Raise rank/learning rate, target more modules, check data formatting.
  • Quality drops after merge — Merge into the full-precision base then quantise, and re-evaluate.

Contrasted with

Questions this page answers

  • What is LoRA?
  • What is QLoRA?
  • How much GPU memory do I need to fine-tune with LoRA?
  • What rank should I use for LoRA?
  • LoRA vs full fine-tuning?

Typical goals: fine-tune with limited GPU memory; manage multiple task adapters.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021) — arXiv:2106.09685
  • Dettmers et al., QLoRA (2023) — arXiv:2305.14314
  • Houlsby et al., Parameter-Efficient Transfer Learning for NLP (2019) — arXiv:1902.00751