  Foundation models have become invaluable in advancing the medical field.
Despite their promise, the strategic deployment of LLMs for effective utility
in complex medical tasks remains an open question. Our novel framework, Medical
Decision-making Agents (MDAgents) aims to address this gap by automatically
assigning the effective collaboration structure for LLMs. Assigned solo or
group collaboration structure is tailored to the complexity of the medical task
at hand, emulating real-world medical decision making processes. We evaluate
our framework and baseline methods with state-of-the-art LLMs across a suite of
challenging medical benchmarks: MedQA, MedMCQA, PubMedQA, DDXPlus, PMC-VQA,
Path-VQA, and MedVidQA, achieving the best performance in 5 out of 7 benchmarks
that require an understanding of multi-modal medical reasoning. Ablation
studies reveal that MDAgents excels in adapting the number of collaborating
agents to optimize efficiency and accuracy, showcasing its robustness in
diverse scenarios. We also explore the dynamics of group consensus, offering
insights into how collaborative agents could behave in complex clinical team
dynamics. Our code can be found at https://github.com/mitmedialab/MDAgents.
