znazz75.de

Fine-Tuning a Small Language Model with LoRA

Claude
,

Full fine-tuning of an LLM means updating every single weight, which for anything beyond a small model requires serious GPU infrastructure. LoRA (Low-Rank Adaptation) instead freezes the original model and trains a tiny set of additional parameters — often under 1% of the original model’s size — making fine-tuning practical on a single consumer GPU.

The Core Idea

Rather than updating a full weight matrix directly, LoRA represents the change to that matrix as the product of two much smaller matrices. Training only those small matrices dramatically reduces both memory usage and the number of trainable parameters, while the original model’s weights stay completely frozen and unchanged.

Fine-Tuning with Hugging Face’s PEFT Library

pip install transformers peft datasets accelerate bitsandbytes

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
from datasets import load_dataset

model_name = "Qwen/Qwen2.5-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

lora_config = LoraConfig(
    r=16,                    # rank: higher = more capacity, more memory
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~2.6M || all params: ~1.5B || trainable%: 0.17%

dataset = load_dataset("json", data_files="my_training_data.jsonl")

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./lora-output",
        per_device_train_batch_size=4,
        num_train_epochs=3,
        learning_rate=2e-4,
    ),
    train_dataset=dataset["train"],
)
trainer.train()
model.save_pretrained("./my-lora-adapter")  # only the small adapter, not the full model

Why This Changes What’s Practical

The saved LoRA adapter is often just a few dozen megabytes, versus gigabytes for the base model — meaning you can maintain many task-specific adapters for the same base model, swap between them at inference time, and train new ones on a single consumer GPU with 16-24GB of VRAM, work that would otherwise require a multi-GPU server.

What LoRA Is Good and Bad At

Excellent for adapting style, tone, and narrow task behavior on top of a capable base model. Not a good fit for teaching entirely new factual knowledge at scale — that’s still RAG’s job, or requires more extensive full fine-tuning.

, , ,

Recommended

Categories