Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System

Researchers proposed Cross-Attention Calibrated Deduplication (CACD) for RAG systems.

CACD filters redundant chunks using cross-encoder comparisons, an attention-entropy-based information score, and majority voting across kept chunks.

On SQuAD 1.1 validation across multiple chunking setups, CACD removed about 9.75% of chunks on average and ran faster than cosine-similarity filtering and the strongest baseline.

The method preserves finer token-level detail while reducing retrieval overhead in RAG pipelines.