The LoRA Assumption That Breaks in Production

Standard LoRA is effective for low-rank style adaptation, but it can struggle to learn distributed factual updates.

As rank increases, the usual 1/r scaling can weaken the learning signal and make high-rank fine-tuning unstable.

RS-LoRA fixes this by using 1/√r scaling, which better preserves update strength at higher ranks.

The result is a more reliable parameter-efficient fine-tuning method for complex factual changes in large models.