Parallax: A Parameterized Local Linear Attention That Keeps Softmax and Adds a Learned Covariance Correction Branch

Researchers introduced Parallax, a new Transformer attention method that keeps softmax while adding a learned local linear covariance correction.

Unlike prior local linear attention methods, Parallax replaces a costly per-query linear solve with a parameterized projection matrix, making it easier to implement and faster to run.

The design preserves the bias-variance benefits of local linear attention while staying compatible with streaming, GPU-friendly execution patterns used by FlashAttention.

The result is a path to better attention quality and higher hardware efficiency at the same time, with relevance for large-scale LLM pretraining on modern GPUs like NVIDIA Hopper.