LoRA Fine Tuning: A Practical Guide

LoRA fine tuning (Low-Rank Adaptation) is a parameter-efficient method for adapting a pretrained language model to a specific task. Instead of updating every weight in the model, LoRA freezes the original weights and injects small trainable matrices into the attention layers. The result is an adapter — a file often under 100 MB — that snaps onto the base model at inference time. You get specialised behaviour without retraining billions of parameters.

How LoRA Works

A standard transformer weight matrix might be 4096 × 4096. LoRA decomposes the update to that matrix into two much smaller matrices: one of shape 4096 × r and one of r × 4096, where r is the rank (commonly 4–64). During training, only these low-rank matrices are updated. At inference, the product of the two small matrices is added back to the frozen original — no extra latency in most implementations, because the adapter is merged before deployment.

This matters for hardware: because the base model weights are frozen, they don’t need optimiser states or gradients. Only the adapter parameters do. That is why VRAM requirements drop so dramatically compared to full fine-tuning.

LoRA vs QLoRA

QLoRA, introduced by Dettmers et al. (2023), loads the base model in 4-bit NormalFloat (NF4) quantisation and keeps the adapter in full precision. Training is slower than standard LoRA because of dequantisation overhead on each forward pass, but the VRAM savings make 13B and 70B models trainable on hardware most people actually own. Quality versus full LoRA is usually negligible for task-specific fine-tunes; for complex reasoning tasks, some degradation is possible.

Before committing to hardware, run your target model through the VRAM calculator — it accounts for batch size and sequence length, both of which move the numbers significantly.

Rank and Alpha: What They Actually Do

Rank (r) controls the expressiveness of the adapter. A rank of 4 adds very few parameters and produces subtle changes; rank 64 gives the adapter more capacity to reshape model behaviour but increases VRAM and overfitting risk.

Alpha (α) is a scaling factor applied to the LoRA output before it is added to the frozen weights. The effective learning rate of the adapter scales with α / r. Keeping alpha at 2× rank (e.g., r=16, alpha=32) is the most common starting point and works well in practice.

Higher rank does not always mean better results. For most instruction-following fine-tunes, r=16 is sufficient. If validation loss is not improving, increase rank or add more data before increasing epochs.

Realistic VRAM and Time Requirements

The figures below assume QLoRA with a batch size of 1 and sequence length of 2048. Multi-GPU setups scale roughly linearly with VRAM but require FSDP or DeepSpeed configuration.

Cloud costs vary. A single A100 80 GB on Lambda Labs runs roughly $1.50–$2.00/hr as of mid-2026. A 7B QLoRA fine-tune on 50k examples typically completes in under two hours — well under $5. For GPU purchase decisions, see the GPU guide for local LLMs.

Tooling: How to Actually Run a LoRA Fine-Tune

The two dominant frameworks are Hugging Face TRL + PEFT and Axolotl. Both support LoRA and QLoRA. Unsloth is a popular third option that achieves 2× faster training via custom CUDA kernels.

Minimal TRL + PEFT Example (Python)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

trainer = SFTTrainer(
    model=model,
    train_dataset=your_dataset,  # expects "text" column
    args=SFTConfig(output_dir="./output", num_train_epochs=3),
)
trainer.train()
model.save_pretrained("./my-lora-adapter")

The adapter saved to ./my-lora-adapter is typically 50–300 MB. Merge it into the base model for faster inference with model.merge_and_unload() before saving.

Axolotl (Config-Driven)

Axolotl drives the whole pipeline from a YAML file, which makes it easier to reproduce runs. Install with pip install axolotl, then:

# config.yml
base_model: meta-llama/Meta-Llama-3-8B-Instruct
load_in_4bit: true
adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
datasets:
  - path: ./data/train.jsonl
    type: alpaca
output_dir: ./output
num_epochs: 3

accelerate launch -m axolotl.cli.train config.yml

When Fine-Tuning Is the Wrong Tool

LoRA fine tuning solves a specific problem: changing how a model behaves or responds. It does not reliably inject factual knowledge. If your use case falls into one of these categories, a different approach will produce better results for less effort:

• The model lacks current or proprietary knowledge. Use retrieval-augmented generation (RAG). Fine-tuning on facts produces models that hallucinate confidently on anything outside the training slice.
• You need the model to follow specific instructions. Try a detailed system prompt first. A well-engineered prompt on a capable base model often outperforms a fine-tuned smaller model on the same task.
• You have fewer than ~500 high-quality examples. The signal-to-noise ratio is too low; the model will likely overfit. Curate more data or use few-shot prompting instead.
• You are prototyping. Fine-tuning locks in a behaviour. Use the API and iterate on prompts until the behaviour is stable, then consider fine-tuning to reduce token costs at scale. The API cost calculator helps quantify when fine-tuning a local model becomes cheaper than paying per token.

If you are weighing a local fine-tuned model against a hosted API long-term, the self-hosting vs API break-even calculator gives you the crossover point based on your volume and GPU costs.

Frequently Asked Questions

How much data do I need for LoRA fine tuning?

For instruction following or style changes, 500–2000 high-quality, diverse examples are often enough. For complex domain adaptation, 5000–20000 examples produce more robust results. Quality matters far more than quantity — 200 carefully curated examples outperform 2000 noisy ones.

Can I run LoRA inference on consumer hardware?

Yes. A merged adapter adds no inference overhead over the base model. An unmerged adapter adds a small amount of computation per forward pass. Both llama.cpp and Ollama support loading GGUF-converted LoRA adapters directly. See the VRAM requirements guide for inference-only memory figures.

What is the difference between LoRA and full fine-tuning?

Full fine-tuning updates every weight in the model and requires storing optimiser states for all of them — roughly 16–20 bytes per parameter in mixed precision with Adam. LoRA updates only the low-rank adapter matrices, reducing trainable parameters by 10–1000×. The trade-off is capacity: full fine-tuning can reshape the model more completely, but for most practical tasks LoRA matches it.

Which layers should I target with LoRA?

The attention projection layers (q_proj and v_proj) are the most common targets and work well for most tasks. Adding k_proj, o_proj, and the MLP layers (gate_proj, up_proj, down_proj) increases capacity at the cost of more VRAM and slightly longer training. If VRAM is tight, start with just q and v projections.

Does QLoRA produce a worse model than full LoRA?

For most task-specific fine-tunes the difference is negligible. Published benchmarks show QLoRA within 1–2 percentage points of full LoRA on standard evals. The gap can widen on complex reasoning tasks with very small datasets, because quantisation noise compounds with limited signal. If accuracy is critical and you have the VRAM, use LoRA over a bf16 base model.

How do I evaluate whether my fine-tune actually helped?

Hold out 10–20% of your data as a validation set and track validation loss during training. Stop when validation loss stops improving (early stopping). Then run task-specific evals: for classification, measure accuracy on held-out examples; for generation, use human review or an LLM-as-judge setup on 50–100 examples. A drop in validation loss that does not translate to better task performance is a sign of distribution mismatch between your training data and real inputs.