NVIDIA Introduces X-Token: Projection-Guided Cross-Tokenizer KD That Outperforms GOLD by 3.82 Average Points on Llama-3.2-1B

NVIDIA introduced X-Token, a drop-in cross-tokenizer knowledge distillation method for models with different tokenizers.

It uses dynamic-programming span alignment, a projection matrix, and complementary losses to transfer knowledge more reliably than prior approaches.

The paper says older methods like GOLD struggle with mismatched or overly strict token matching.

X-Token reportedly beats GOLD by 3.82 average points on Llama-3.2-1B and improves cross-tokenizer KD for smaller student models.