Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

Researchers introduced RTPurbo, a sparse inference method for full-attention LLMs that keeps the full KV cache only for retrieval heads.

It uses a lightweight dynamic sparse token indexer to exploit intrinsic attention sparsity in long contexts.

With only a few hundred adaptation steps, it achieves near-lossless accuracy on long-context benchmarks.

The approach can significantly speed up both prefill and decoding without costly sparse pretraining.