Delta Attention Residuals

Researchers propose Delta Attention Residuals, a transformer routing method that attends to layer-to-layer changes instead of cumulative hidden states.

They find standard Attention Residuals become overly uniform across layers, reducing contrast in attention weights.

Routing over deltas produces sharper attention distributions and lowers validation perplexity across models from 220M to 7.6B parameters.

The method also offers a practical way to improve pretrained checkpoints by fine-tuning layer-wise routing.