TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks

Researchers introduced TerminalWorld, a benchmark derived from 80,870 public human terminal recordings and reverse-engineered 1,530 validated tasks.

The benchmark spans 18 categories and 1,280 tools or commands, aiming to capture messy, real-world terminal workflows.

When frontier models were tested, the best result was only a 62.5% pass rate, showing substantial room for improvement.

The findings suggest current AI agents still struggle with practical terminal work in environments like Docker, CI/CD, cloud, and system administration.