  Multimodal artificial intelligence (AI) systems have the potential to enhance
clinical decision-making by interpreting various types of medical data.
However, the effectiveness of these models across all medical fields is
uncertain. Each discipline presents unique challenges that need to be addressed
for optimal performance. This complexity is further increased when attempting
to integrate different fields into a single model. Here, we introduce an
alternative approach to multimodal medical AI that utilizes the generalist
capabilities of a large language model (LLM) as a central reasoning engine.
This engine autonomously coordinates and deploys a set of specialized medical
AI tools. These tools include text, radiology and histopathology image
interpretation, genomic data processing, web searches, and document retrieval
from medical guidelines. We validate our system across a series of clinical
oncology scenarios that closely resemble typical patient care workflows. We
show that the system has a high capability in employing appropriate tools
(97%), drawing correct conclusions (93.6%), and providing complete (94%), and
helpful (89.2%) recommendations for individual patient cases while consistently
referencing relevant literature (82.5%) upon instruction. This work provides
evidence that LLMs can effectively plan and execute domain-specific models to
retrieve or synthesize new information when used as autonomous agents. This
enables them to function as specialist, patient-tailored clinical assistants.
It also simplifies regulatory compliance by allowing each component tool to be
individually validated and approved. We believe, that our work can serve as a
proof-of-concept for more advanced LLM-agents in the medical domain.
