Understanding Data Temporality Impact on Large Language Models Pre-training

A new study shows that the order of pre-training data can matter for LLMs.

Researchers benchmarked 7,000+ temporally grounded questions and trained 6B-parameter models on ordered vs. shuffled web data.

Models trained sequentially were better at temporal precision and factual freshness, while maintaining general language performance.

The findings suggest data ordering can help build more time-aware, up-to-date language models.