How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Researchers modeled exact parametric memory in LLM fine-tuning with LoRA and found memory loss follows a power-law relationship with model and training factors.

They identified a token-level recall threshold that predicts when verbatim memory starts to fail under greedy decoding.

The study introduces MemFT, a threshold-guided training method that focuses on hard-to-learn tokens.

MemFT improves memory fidelity and makes exact recall more efficient for large language models.