Linearizing Vision Transformer with Test-Time Training

Researchers propose a way to convert pretrained Vision Transformers from softmax attention to linear-complexity test-time training models.

The method aligns both architecture and representations, using key instance normalization and a locality module to inherit weights with limited fine-tuning.

On Stable Diffusion 3.5, the fine-tuned SD3.5-T^5 variant matches quality while running faster, cutting inference cost on H20 GPUs.