Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

Researchers introduced AutoTool, a multimodal LLM that learns when to use external tools and when to reason without them.

The paper finds that redundant tool calls can hurt performance, so adaptive tool invocation can be better than always using tools.

AutoTool uses reinforcement learning to choose between tool-assisted and text-only reasoning on the fly.

Reported results show strong gains in both accuracy and efficiency on benchmarks such as V* and POPE.