Predictive Divergence Masks for LLM RL

The paper proposes a predictive divergence mask for LLM reinforcement learning that uses the same divergence as the trust-region check to decide whether to mask a policy update.

This replaces PPO’s sampled-token ratio-based direction test with a more consistent divergence-based criterion.

The authors derive a closed-form version for discrete softmax policies and add top-K estimators for rollout systems with truncated vocabularies.

The method improves training stability and consistency across model sizes and precision settings by aligning masking decisions with policy-drift control.