For makers and artists working with open models, the default choice of Low Rank Adaptation (LoRA) might be holding you back. While it is the undisputed king of parameter-efficient fine-tuning (PEFT), relying on it blindly means you could be leaving performance on the table. The landscape is shifting, and newer techniques can offer better accuracy or lower memory footprints depending on your specific needs.
Why you need PEFT
Open models are abundant, yet often insufficient for niche tasks. While prompting offers a workaround, it rarely delivers the nuance required. Training from scratch is prohibitively expensive, so fine-tuning existing weights is the logical step. However, full fine-tuning is memory-intensive, often requiring several copies of the model in VRAM. Quantisation helps reduce footprint, but these models cannot be trained directly. PEFT solves this by allowing you to train a fraction of the model, enabling even quantised versions to be updated efficiently. This approach yields smaller checkpoints, reduces catastrophic forgetting, and allows multiple specialised models to run from a single base.
Hugging Face’s PEFT library unifies these methods behind a single API, integrating seamlessly with ecosystems like Transformers and Diffusers. It supports various quantisation methods, making experimentation accessible for both practitioners and researchers.
LoRA’s dominance and the problem of popularity
Low Rank Adaptation (LoRA) works by freezing base weights and training only a small set of added parameters. Its ubiquity is staggering:
- Of 20,834 model cards on Hugging Face Hub mentioning a single PEFT technique, 20,509 (98.4%) used LoRA.
- Among 10,000 image generation checkpoints, 7,111 were LoRAs, leaving only 95.0% for other methods like LoCon and DoRA.
- GitHub searches for
from peft importshow 71.3% of results are LoRA, with LoHa and AdaLoRA trailing significantly.
This dominance suggests two possibilities. First, LoRA is simply the best tool for everyone. Second, its early adoption created a self-reinforcing cycle of visibility, tutorials, and support. Consequently, many developers stick with the default, potentially ignoring superior alternatives that researchers have published.
The trap of paper benchmarks
Dozens of papers claim their methods outperform LoRA, yet relying on these claims is fraught with issues. Researchers often face pressure to beat existing benchmarks, leading to biased comparisons where the proposed method is tuned more rigorously than the controls. For instance, one study demonstrated that LoRA could match supposedly superior techniques simply by adjusting the learning rate.
Furthermore, papers frequently compare disparate techniques against arbitrary benchmarks, and the code is often unavailable or difficult to reproduce. This opacity makes it nearly impossible to determine the best method for your specific dataset without running your own tests.
A fair benchmarking approach
To address this, Hugging Face built a benchmark suite within the PEFT library that evaluates techniques on equal footing. The suite includes:
- LLM Math Benchmark: Tests chain-of-thought reasoning on the MetaMathQA dataset to see if models can learn mathematical logic and format output correctly.
- Image Generation Benchmark: Evaluates the ability to learn a new concept (e.g., a cat plushy) without forgetting existing knowledge, measured by DINO similarity.
All experiments run on identical hardware, using the same base models, datasets, and code. Beyond simple test accuracy, the suite tracks VRAM usage, runtime, checkpoint size, and forgetting metrics. This objective data allows users to see the true tradeoffs rather than relying on cherry-picked paper results.
Findings: LoRA is good, but not always best
The benchmarks reveal that while LoRA is strong, it is not always on the Pareto Frontier-the set of optimal solutions where you cannot improve one metric without worsening another.
LLM Math Results:
LoRA achieves 53.2% accuracy with 22.6 GB of peak VRAM. However, other methods offer different tradeoffs. BEFT uses less memory (20.2 GB) for lower accuracy (32.9%), while Lily offers higher accuracy (54.9%) at the cost of more VRAM (25.6 GB). Crucially, standard LoRA is not the peak performer here; variants like LoRA with rank-stabilised initialization or LoRA-FA (which freezes some LoRA weights) outperform the vanilla version, which only hits 48.1% accuracy.
Image Generation Results:
When fine-tuning FLUX.2-klein-base-4B to recognise a cat plushy, LoRA scored 0.697 on DINO similarity using 9.97 GB of VRAM. The method OFT strictly dominated LoRA, achieving a higher similarity score of 0.708 while using less memory (9.01 GB). This proves that for specific tasks, newer techniques can offer both better fidelity and efficiency.
The takeaway is clear: metrics like runtime or checkpoint size can shift the optimal choice. Users must inspect the specific tradeoffs relevant to their workflow rather than defaulting to the most popular tool.
Limitations
Objection: But the benchmarks favour one method over another!
While hyperparameter choices can influence outcomes, the benchmarking framework is designed to be neutral. By enforcing identical training conditions and hardware, the suite aims to remove the bias often found in academic papers. The goal is to provide a transparent starting point for users to make informed decisions.
Key takeaways
- Don’t default to LoRA: While popular, LoRA is not always the most efficient or accurate choice for your specific dataset or hardware constraints.
- Look for the Pareto Frontier: Better performance often comes at a memory cost, and vice versa; tools like BEFT or OFT may offer a better tradeoff depending on your priorities.
- Run your own benchmarks: Use the
PEFTlibrary to test multiple techniques on your own data to find the optimal configuration for your use case.
