Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs

A controlled benchmark evaluated 13 attribution methods on eight vision backbones, including CNNs and transformer-based models, using faithfulness, localization, robustness, complexity, and cost.

Performance proved strongly architecture-dependent: rankings that look good on CNNs often do not transfer to Vision Transformers or related variants.

CAM-style methods score well on standard localization metrics, but those metrics can saturate without truly precise localization; attention rollout is robust but weaker at localization.

The study argues that faithfulness metrics alone do not cleanly distinguish methods, so XAI evaluation should be multi-metric and architecture-specific.