Towards Efficient LLMs Annealing with Principled Sample Selection

Researchers study the annealing phase of LLM pre-training using loss-landscape spectral geometry and Hessian eigen-directions.

They introduce DiReCT, a curvature-aware sample selection method that enforces directional gradient constraints on training samples.

Across model scales, DiReCT shows strong results, suggesting better convergence and final performance.

The work provides a theory-grounded approach to data selection that could improve both training efficiency and model quality.