Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Memory Decoder at Scale pretrains a long-term memory module for decoder-only language models at 6.9B parameters on 300B tokens.

The team also built a distributed Faiss indexing and retrieval pipeline to support large-scale memory search.

Across model sizes and benchmarks, giving more parameters to memory improved the performance-to-parameter tradeoff.

The findings suggest that scaling memory separately can raise LM quality with fewer total parameters than enlarging the base model alone.