By Ryze AI Team
LoRA, Low-Rank Adaptation, is the standard way to specialize a model without the cost of full fine-tuning. Instead of updating every weight, it trains a small set of adapter matrices that are added to selected layers. The base model stays frozen, memory and compute drop by an order of magnitude or more, and the result is a compact adapter file that can be swapped in and out.
Fine-tune when prompting and RAG are not enough. Good reasons: enforcing a specific output style or format the base model mishandles, adapting to a narrow domain vocabulary, or teaching a small model a task it cannot do at all through prompting. Bad reasons: adding facts (use RAG), fixing a single wrong answer (fix the prompt), or chasing a benchmark. Fine-tuning is a commitment; it requires data, evaluation, and ongoing maintenance as the base model changes.
For a weight matrix W, LoRA adds a low-rank update: W becomes W plus B times A, where A and B are small matrices whose product has rank r, much smaller than the original dimensions. Only A and B are trained. The rank r controls capacity: higher r means more expressiveness and more parameters. In practice, adapters are applied to the attention projections (query, key, value, output) and sometimes the feed-forward layers. The base model's weights are frozen throughout, which is what makes training cheap and prevents catastrophic forgetting of general capabilities.
Quality beats quantity for small models. Five hundred to two thousand clean, diverse examples often suffice. Format them as instruction-input-output triples, or as chat messages with system, user, and assistant turns. Deduplicate, balance classes, and remove near-duplicates that would leak between train and eval. A common failure is training on noisy or inconsistent labels, which teaches the model to be inconsistent. Spend more time on data than on hyperparameters.
Rank 8 to 32 is the typical range; start at 16. Alpha is usually set equal to rank or twice it, and controls how much the adapter influences the base model. Learning rate of 1e-4 to 2e-4 works well. Train one to three epochs; more risks overfitting, which shows up as rising eval loss or degraded general behavior. Use a warmup and cosine schedule. Monitor both task-specific eval loss and a general-capability eval to catch forgetting early.
You have two options. Load the adapter dynamically alongside the base model, which keeps the base model shared across multiple adapters but adds a small runtime overhead. Or merge the adapter into the base weights with merge_and_unload, producing a single model that serves like any other. Merging is simpler for serving and is what you usually want for production. After merging, quantize normally. Always re-run your eval set on the merged, quantized model, because each step can shift behavior.
Resources: PEFT and Unsloth, which speeds up training considerably.