DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Researchers introduced DataPrep-Bench, a benchmark for evaluating how well LLMs prepare training data from raw sources.

It covers two tasks: constructing training data and scoring candidate datasets by how well they improve downstream performance.

The paper also proposes Data-Construction-Skill for data generation and DAS, or Distributional Alignment Score, for dataset quality assessment.

The benchmark aims to unify comparisons of data-generation and data-selection methods based on actual utility for training LLMs.