PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

PEFT evaluation is reframed to include both downstream adaptation and forgetting of pretrained skills, not just task accuracy.

The paper introduces PEFT-Arena, a benchmark that measures task performance alongside general-capability retention in large language models.

Across similar parameter budgets, orthogonal finetuning shows the best stability-plasticity balance and strongest overall trade-off.

The study explains method differences with weight-space and activation-space geometry, and proposes path-wise rewinding as a post-hoc boost.