Multi-Turn On-Policy Distillation with Prefix Replay

Researchers proposed ReOPD, an offline on-policy distillation method for multi-turn LLM agents that reuses stored teacher prefixes instead of new environment rollouts.

The paper identifies a “prefix trap,” where student-generated histories drift from reliable teacher trajectories and make multi-turn distillation harder.

ReOPD uses replayed prefixes plus a step-decaying sampling strategy to reduce this mismatch across turns.

In reasoning and search tasks, the method matches or improves standard on-policy distillation while avoiding tool calls and speeding training.

The approach could make agent distillation much cheaper and easier to scale across tasks and model sizes.