New math benchmark reveals AI models confidently solve problems that have no solution

SOOHAK, a new 439-problem math benchmark, tests both graduate-level problem solving and intentionally flawed refusal cases.

Top frontier models performed well below human experts on the hardest tasks, showing limits in advanced mathematical reasoning.

No model scored above 50% on spotting unsolvable or invalid questions, exposing weak refusal and error-detection behavior.

The results suggest that scaling AI improves more routine math, but not yet reliable judgment about when a problem has no valid answer.