From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

Researchers introduced SubFit, a post-training compression method for large language models that works at the submodule level.

Instead of removing whole layers, SubFit selects Attention and FeedForward blocks non-contiguously and replaces them with fitted residual bypasses.

Across ten models and multiple sparsity settings, it beat several replacement-based baselines, especially at higher compression levels.

The method also delivered speed and memory gains, including inference speedup and KV-cache savings.

Overall, SubFit points to a finer-grained way to compress transformers while preserving more accuracy.