SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation

Researchers introduced SegCompass, an interpretable segmentation model that aligns chain-of-thought reasoning with visual concepts using a sparse autoencoder-based concept space.

The system maps text reasoning and image tokens into shared sparse concepts, then uses a query codebook and slot mapper to generate spatial heatmaps for mask decoding.

SegCompass is trained end to end with reinforcement learning and segmentation supervision, aiming to keep reasoning and visual output more transparent and inspectable.

On five benchmarks, it delivers competitive or better segmentation results, with stronger learned concepts associated with higher mask accuracy.