Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

Researchers introduced SemanticQA, a unified benchmark built from multiword expression resources.

It tests language models on idioms, noun compounds, verbal constructions, and collocations.

The benchmark measures extraction, classification, interpretation, and task composition in semantic reasoning.

SemanticQA helps reveal weaknesses in language understanding that simpler benchmarks may miss.