  The reasoning capabilities of LLM (Large Language Model) are widely
acknowledged in recent research, inspiring studies on tool learning and
autonomous agents. LLM serves as the "brain" of the agent, orchestrating
multiple tools for collaborative multi-step task solving. Unlike methods
invoking tools like calculators or weather APIs for straightforward tasks,
multi-modal agents excel by integrating diverse AI models for complex
challenges. However, current multi-modal agents neglect the significance of
model selection: they primarily focus on the planning and execution phases, and
will only invoke predefined task-specific models for each subtask, making the
execution fragile. Meanwhile, other traditional model selection methods are
either incompatible with or suboptimal for the multi-modal agent scenarios, due
to ignorance of dependencies among subtasks arising by multi-step reasoning. To
this end, we identify the key challenges therein and propose the $\textit{M}^3$
framework as a plug-in with negligible runtime overhead at test-time. This
framework improves model selection and bolsters the robustness of multi-modal
agents in multi-step reasoning. In the absence of suitable benchmarks, we
create MS-GQA, a new dataset specifically designed to investigate the model
selection challenge in multi-modal agents. Our experiments reveal that our
framework enables dynamic model selection, considering both user inputs and
subtask dependencies, thereby robustifying the overall reasoning process. Our
code and benchmark: https://github.com/LINs-lab/M3.
