# LoRA and QLoRA: Fine-Tuning Without a GPU Farm

> Train a few million adapter parameters instead of billions, and when fine-tuning beats prompting or RAG.

- Source: https://akshaythakare.com/notes/llms/lora-fine-tuning
- Topic: LLMs
- Level: advanced
- Reading time: 3 min
- Tags: llms, fine-tuning, lora, qlora, peft
- Updated: 2026-10-11

## Full fine-tuning is expensive

Updating every weight of a 7-billion-parameter model means storing gradients and optimiser state for all of them — tens of gigabytes of GPU memory, and a full copy of the model per task.

## The LoRA idea

Freeze the original weights. For selected weight matrices `W`, learn a **low-rank update** `ΔW = B·A`, where `A` and `B` are thin matrices with rank `r` (often 8–64).

```text
W  : 4096 × 4096  = 16.8M params (frozen)
A  : r × 4096, B : 4096 × r   with r = 16  →  131k trainable params
output = W·x + (B·A)·x · (alpha / r)
```

**LoRA training:** Base model (frozen weights) → Adapters (A·B, rank r) → Training data (instruction pairs) → Train adapters (few million params) → Merge or load (per task)

## QLoRA: also shrink the base model

QLoRA loads the frozen base model in **4-bit** precision and trains LoRA adapters on top. A 7B model then fits comfortably on a single consumer GPU.

```python
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

base = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"),
)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05))
model.print_trainable_parameters()   # ~0.1% of the model
```

## Prompting, RAG or fine-tuning?

  
### Try prompting first
Clear instructions and examples solve most formatting and tone problems.
  
### Use RAG for knowledge
If the model needs your documents or fresh data, retrieve them — don't bake facts into weights.
  
### Fine-tune for behaviour
Consistent output structure, domain style, or a narrow skill like NL-to-SQL on your schema.
  
### Evaluate before and after
Hold out a test set; fine-tuning can quietly hurt general ability.