Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

Researchers introduced SAE-FT, a sparse autoencoder-based method for fine-tuning CLIP.

It regularizes visual features without costly text-guidance, helping reduce catastrophic forgetting.

The approach improves robustness and interpretability while matching or beating strong results on ImageNet and distribution-shift benchmarks.