LoRA and QLoRA: Fine-Tuning Without a GPU Farm

Full fine-tuning is expensive

Updating every weight of a 7-billion-parameter model means storing gradients and optimiser state for all of them — tens of gigabytes of GPU memory, and a full copy of the model per task.

The LoRA idea

Freeze the original weights. For selected weight matrices W, learn a low-rank update ΔW = B·A, where A and B are thin matrices with rank r (often 8–64).

W  : 4096 × 4096  = 16.8M params (frozen)
A  : r × 4096, B : 4096 × r   with r = 16  →  131k trainable params
output = W·x + (B·A)·x · (alpha / r)

QLoRA: also shrink the base model

QLoRA loads the frozen base model in 4-bit precision and trains LoRA adapters on top. A 7B model then fits comfortably on a single consumer GPU.

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
base = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"),
)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05))
model.print_trainable_parameters()   # ~0.1% of the model

Prompting, RAG or fine-tuning?

1. 1

   Try prompting first

   Clear instructions and examples solve most formatting and tone problems.
2. 2

   Use RAG for knowledge

   If the model needs your documents or fresh data, retrieve them — don't bake facts into weights.
3. 3

   Fine-tune for behaviour

   Consistent output structure, domain style, or a narrow skill like NL-to-SQL on your schema.
4. 4

   Evaluate before and after

   Hold out a test set; fine-tuning can quietly hurt general ability.