Language Models Need Sleep

Researchers propose a sleep-like consolidation method for long-context language models.

The model periodically compresses recent context into persistent fast weights during a sleep phase, then clears the cache and resumes normal-speed inference.

On synthetic and reasoning benchmarks, longer sleep phases improve performance, especially on deeper reasoning tasks.

The approach aims to ease attention scaling limits by boosting memory and reasoning without increasing wake-time latency.