A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

Researchers introduced A2RBench, an automated pipeline for generating and expanding abstract reasoning tasks with programmatic verification.

The benchmark is designed to ensure unique solutions and scalable, verifiable evaluation beyond memorization.

Results show mainstream LLMs still trail humans on abstract reasoning, with especially weak performance on 3D tasks.

The work highlights a practical way to stress-test reasoning and expose gaps in today’s top models.