Applied LLMs
DoRA: Weight-Decomposed LoRA
DoRA decomposes pre-trained weights into magnitude and direction components, then applies LoRA exclusively to the directional part, closing most of the accuracy gap between LoRA and full fine-tuning without adding inference overhead.
advanced · 7 min read · Premium
Standard LoRA with rank 16 on LLaMA-7B reaches 74.7% average accuracy across eight commonsense reasoning benchmarks. Full fine-tuning reaches roughly 79%. DoRA, with the same rank and the same parameter budget, reaches 78.4% - recovering most of that gap by changing not what parameters are trained, but how the weight update is structured.
That five-point gap is not a minor tuning artefact. It traces to a fundamental constraint in LoRA's update geometry: magnitude and direction are forced to move together, whereas full fine-tuning moves them independently. DoRA was designed to remove that constraint.
What LoRA Gets Wrong About Weight Updates
Keep reading with Pro.
You're reading the preview. Unlock the full concept plus the library, study plans, the AI mentor, and daily emails.