UCSD and Together AI Researchers Introduce Parcae: A Stable Architecture for Looped Language Models That Achieves the Quality of a Transformer Twice the Size

UC San Diego and Together AI introduced Parcae, a middle-looped Transformer with built-in stability constraints.

By treating the loop as a dynamical system and constraining the residual update, it avoids the training instability seen in earlier looped models.

At tested scales, it outperforms matched baselines, including fixed-depth Transformers and prior recurrent depth models, with the same parameter budget.

The approach can improve model quality without adding parameters or data, supporting cheaper inference and edge deployment.