AI models fail at robot control without human-designed building blocks but agentic scaffolding closes the gap

CaP-X is an open-access framework released by researchers from Nvidia, UC Berkeley, Stanford, and Carnegie Mellon that systematically evaluates AI coding agents controlling robots via self-written programs.

The study tested twelve frontier models (including Gemini-3-Pro, GPT-5.2, Claude Opus 4.5, Qwen3-235B, DeepSeek-V3.1) across seven manipulation tasks and found none matched the reliability of human-written programs in a single attempt.

The core idea contrasts with training robot-specific motion models: general-purpose language models generate the control code instead of using motion datasets.

Techniques adapted from language-model work include reinforcement learning with verifiable physics-simulation rewards, test-time compute scaling via parallel solution generation and self-correction, and agentic patterns like automated debugging and accumulating reusable functions.

Results depend on available building blocks: pre-built commands improve performance, while replacing them with low-level steps (image segmentation, depth processing, grasp planning, inverse kinematics) or feeding raw camera images into the model context lowers success rates, linked to limited multimodal joint-reasoning training and cross-modal alignment issues.