Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Researchers found that small scale vectors in LLM normalization layers have a real impact on pretraining, despite their tiny size.

In Pre-Norm models, the vectors mostly help optimization rather than model expressivity, while weight decay helps Input-Norm but hurts Output-Norm layers.

Three lightweight changes—branch-specific heterogeneity, better placement near linear layers, and magnitude-direction reparameterization—consistently improved training.

The combined approach worked across dense and mixture-of-experts models from 0.12B to 2B parameters, lowering terminal loss with little extra cost.