  Modern medical records include a vast amount of multimodal free text clinical
data and imaging data from radiology, cardiology, and digital pathology. Fully
mining such big data requires multitasking; otherwise, occult but important
aspects may be overlooked, adversely affecting clinical management and
population healthcare. Despite remarkable successes of AI in individual tasks
with single-modal data, the progress in developing generalist medical AI
remains relatively slow to combine multimodal data for multitasks because of
the dual challenges of data curation and model architecture. The data challenge
involves querying and curating multimodal structured and unstructured text,
alphanumeric, and especially 3D tomographic scans on an individual patient
level for real-time decisions and on a scale to estimate population health
statistics. The model challenge demands a scalable and adaptable network
architecture to integrate multimodal datasets for diverse clinical tasks. Here
we propose the first-of-its-kind medical multimodal-multitask foundation model
(M3FM) with application in lung cancer screening and related tasks. After we
curated a comprehensive multimodal multitask dataset consisting of 49 clinical
data types including 163,725 chest CT series and 17 medical tasks involved in
LCS, we develop a multimodal question-answering framework as a unified training
and inference strategy to synergize multimodal information and perform multiple
tasks via free-text prompting. M3FM consistently outperforms the
state-of-the-art single-modal task-specific models, identifies multimodal data
elements informative for clinical tasks and flexibly adapts to new tasks with a
small out-of-distribution dataset. As a specialty-oriented generalist medical
AI model, M3FM paves the way for similar breakthroughs in other areas of
medicine, closing the gap between specialists and the generalist.
