OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

Researchers introduced OmniGUI, a step-level multimodal benchmark for smartphone GUI agents.

The benchmark uses interleaved images, audio, and video to reflect real mobile context more accurately than screenshot-only tests.

It includes 709 expert-demonstrated episodes across 29 apps.

Baseline results show agents perform worse when decisions require synchronized temporal and auditory signals.