Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

Researchers introduced a mid-training stage where language models generate and filter multiple correct solutions before reinforcement learning.

Training on diverse self-generated reasoning traces improves GRPO-based RL versus vanilla RL and STaR+RL across multiple benchmarks.

Gains are larger at higher pass@k, suggesting richer priors help more when the model can explore several good paths.

Analysis shows RL increasingly composes multiple heuristics over time instead of relying on a single memorized solution path.