Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models

Researchers introduced TIDE, a cross-architecture distillation framework for diffusion large language models.

TIDE uses three components to transfer knowledge from larger teachers to smaller students with different tokenizers and attention designs.

On eight benchmarks, the 0.6B student beat baseline results and delivered major gains in efficiency versus the 16B teacher.

The work suggests diffusion LLMs can be made more practical through better accuracy, code generation, memory use, and inference speed.

Code, checkpoints, and data have been released publicly by the PKU-YuanGroup team.