Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation

Researchers introduced MetaFine, a diagnostic evaluation framework for fine-grained robot manipulation that decomposes tasks into understanding, perception, and control.

It converts existing manipulation benchmarks into graded diagnostic tasks to test multiple capability axes more precisely than binary success metrics.

Across state-of-the-art vision-language-action models, MetaFine finds local spatial perception as a major bottleneck.

The study also adds a hybrid real-simulation validation method to make benchmarking more reliable and calibrated.