AI Foundation Model Enhances Cancer Diagnosis and Tailors Treatment
A new study and AI model from researchers at Stanford University is streamlining cancer diagnostics, treatment planning, and prognosis prediction. Named MUSK (Multimodal transformer with Unified maSKed modeling), the research
aims to advance precision oncology, tailoring treatment plans to each patient based on their unique medical data.
“Multimodal foundation models are a new frontier in medical AI research,” said
Ruijiang LI
, an associate professor of radiation oncology and study senior author. “Recently, vision–language foundation models have been developed for medicine, particularly in the field of pathology. However, existing studies use off-the-shelf foundation models that require paired image–text data for pretraining. Despite extensive efforts that led to the curation of 1M pathology image–text pairs, it’s still insufficient to fully capture the diversity of the entire disease spectrum.”
Oncologists rely on many data sources when considering a patient’s condition and planning optimal treatments. However, integrating and interpreting complex medical data remains difficult for doctors and AI models. The
study
, recently published in
Nature
, highlights how MUSK could help doctors make more accurate and informed decisions while also solving this long-standing challenge in medical AI.
Using deep learning, MUSK processes clinical text data (such as doctor’s notes) and pathology images (like histology slides), to identify patterns that may not be immediately obvious to doctors, leading to better clinical insights.
To do so, it uses a two-step multimodal transformer model. First, it learns from large amounts of unpaired data, pulling features from the text and images that are useful. Then it finetunes its understanding of the data by linking paired image-text data, which helps it recognize different types of cancer, predict biomarkers, and suggest effective treatment options.
The researchers pretrained the AI model on one of the biggest datasets in the field, using 50M pathology images from 11,577 patients with 33 tumor types and 1B pathology-related text data.
According to
Jinxi Xiang
, study lead author and postdoctoral scholar in radiation physics, the pretraining was conducted over 10 days using 64
NVIDIA V100 Tensor Core
GPUs across eight nodes, enabling MUSK to process vast amounts of pathology images and clinical text efficiently. A secondary pretraining phase and ablation studies used
NVIDIA A100 80 gb Tensor Core GPUs
. The researchers also used
NVIDIA RTX A6000 GPUs
for evaluating downstream tasks. The framework was accelerated with
NVIDIA CUDA
and
NVIDIA cuDNN
libraries, for optimized performance.
When tested on 23 pathology benchmarks, MUSK outperformed existing AI models in several key areas. It excelled at matching pathology images with correlating medical text, making it more effective in gathering relevant patient information. It also interpreted pathology-related questions, such as identifying a cancerous area or predicting biomarker presence with 73% accuracy.
Figure 1. An example of the visual question-answering MUSK can perform
It improved detection and classification for cancer subtypes including breast, lung, and colorectal cancer by up to 10%, which could help with early diagnosis and treatment planning. It also detected ‌breast cancer biomarkers with an AUC (a measure of model accuracy) of 83%.
Additionally, MUSK reliably predicted cancer survival outcomes 75% of the time, and which lung and gastro-esophageal cancers would respond to immunotherapy with 77% accuracy. This is a significant improvement over standard clinical biomarkers with an accuracy of only 60-65%.
“One striking finding is that AI models that integrate multi-modal data consistently outperform those based on imaging or text data alone, highlighting the power of a multimodal approach,” Li said. “The true value of MUSK lies in its ability to leverage large-scale unpaired image and text data for pretraining, which is a substantial increase over existing models that require paired data.”
A core strength of the research is that it can adapt across different clinical settings with little training. This could improve efficiency in oncology workflows and help doctors diagnose cancer faster while tailoring treatments for better patient outcomes.
Their future work will focus on validating the model in multi-institution cohorts of patients from diverse populations and for high-stakes applications such as treatment decision-making. The researchers note that prospective validation in clinical trials will be required for regulatory approval.
“We are also working on an extension of the MUSK approach to digital pathology to other types of data such as radiology images and genomic data,” said Li.
The researchers’ work, including installation instructions, model weights, evaluation code, and sample data is available on
GitHub
.
AI 基础模型增强癌症诊断并实现个性化治疗
斯坦福大学研究人员的一项新研究和 AI 模型正在简化癌症诊断、治疗规划和预后预测。这项名为 MUSK (Multimodal transformer with Unified maSKed modeling) 的研究旨在提高精准肿瘤学，根据每位患者独特的医疗数据为其定制治疗计划。
“多模态基础模型是医学 AI 研究的新领域，”放射肿瘤学副教授兼研究高级作者
Ruijiang LI
说。“最近，我们为医学领域开发了视觉语言基础模型，尤其是在病理学领域。但是，现有研究使用的现有基础模型需要配对的图像 – 文本数据进行预训练。尽管我们付出了大量努力，最终打造出 1M 病理图像文本对，但它仍然不足以完全捕捉整个疾病谱系的多样性。”
在考虑患者状况和规划最佳治疗方案时，肿瘤科医生依靠多种数据源。然而，医生和 AI 模型仍然难以集成和解释复杂的医疗数据。该研究最近发表在 Nature 杂志上，重点介绍了 MUSK 如何帮助医生做出更准确、更明智的决定，同时解决医学 AI 领域的长期挑战。
借助深度学习，MUSK 处理临床文本数据（如医生的笔记）和病理学图像（如组织学幻灯片），以识别医生可能无法立即发现的模式，从而获得更好的临床见解。
为此，它使用了两步多模态 transformer 模型。首先，它从大量未配对的数据中学习，从有用的文本和图像中提取特征。然后，它通过关联配对的图像-文本数据来微调对数据的理解，这有助于识别不同类型的癌症、预测生物标志物，并提出有效的治疗方案。
研究人员基于该领域最大的数据集之一预训练了 AI 模型，使用了来自 11,577 名患者的 50M 病理学图像，其中有 33 种肿瘤类型和 1B 病理学相关文本数据。
据辐射物理学研究主要作者兼博士后学者 Jinxi Xiang 称，预训练在 8 个节点上使用 64 个
NVIDIA V100 Tensor Core
GPUs 进行了 10 天以上，使 MUSK 能够高效处理大量病理学图像和临床文本。二级预训练阶段和消融研究使用
NVIDIA A100 80GB Tensor Core GPUs
。研究人员还使用
NVIDIA RTX A6000 GPUs
评估下游任务。该框架通过
NVIDIA CUDA
和
NVIDIA cuDNN
库进行加速，以优化性能。
在 23 项病理学基准测试中，MUSK 在多个关键领域的表现优于现有 AI 模型。它擅长将病理学图像与相关的医学文本进行匹配，从而更有效地收集相关的患者信息。它还能解读与病理学相关的问题，例如识别癌变区域或预测生物标志物的存在，准确率高达 73%。
图 1. 例如，视觉问答 MUSK 可以执行
它将乳腺癌、肺癌和结直肠癌等癌症亚型的检测和分类能力提高了 10%，这有助于早期诊断和治疗规划。它还检测到乳腺癌生物标志物，AUC（用于衡量模型准确性的指标）为 83%。
此外，MUSK 有 75%的时间能够可靠预测癌症生存期结果，以及哪些肺癌和胃食道癌会对免疫治疗做出反应，准确率为 77%。与准确率仅为 60-65%的标准临床生物标志物相比，这是一个显著的改进。
“一个惊人的发现是，集成多模态数据的 AI 模型的性能始终优于仅基于图像或文本数据的 AI 模型，这凸显了多模态方法的强大功能，”Li 说。“MUSK 的真正价值在于它能够利用大规模的未配对图像和文本数据进行预训练，与需要配对数据的现有模型相比，这是一个巨大的提升。”
这项研究的一个核心优势是，它可以在几乎没有训练的情况下适应不同的临床环境。这可以提高肿瘤学工作流程的效率，并帮助医生更快地诊断癌症，同时定制治疗方案以改善患者的治疗效果。
他们未来的工作重点将是在来自不同人群的多机构患者群体中验证该模型，以及用于治疗决策等高风险应用。研究人员指出，临床试验中的前瞻性验证需要获得监管机构的批准。
“我们还致力于将 MUSK 方法扩展到数字病理学，包括放射学图像和基因组数据等其他类型的数据，”Li 说。
研究人员的工作（包括安装说明、模型权重、评估代码和样本数据）
可在 GitHub 上获取。