Enhancing Rubric-based RL via Self-Distillation

Researchers identify two weak points in rubric-based RL for LLMs: criteria that never get optimized and criteria whose positive signal gets suppressed.

They propose Criterion-Distilled Policy Optimization (CriPO), which uses on-policy self-distillation with two self-teachers.

A criterion-injection self-teacher learns missing behaviors, while a counterfactual self-teacher restores suppressed token-level advantages.

On medicine and science benchmarks, CriPO delivers better performance with about half the optimization steps.

The method improves open-ended LLM training without relying on external rollout guidance that can cause inference-time mismatch.