  Mapping computed tomography (CT) number to material property dominates the
proton range uncertainty. This work aims to develop a physics-constrained deep
learning-based multimodal imaging (PDMI) framework to integrate physics, deep
learning, magnetic resonance imaging (MRI), and advanced dual-energy CT (DECT)
to derive accurate patient mass density maps. Seven tissue substitute MRI
phantoms were used for PDMI-based material calibration. The training inputs are
from MRI and twin-beam dual-energy images acquired at 120 kVp with gold and tin
filters. The feasibility investigation included an empirical DECT correlation
and four residual networks (ResNet) derived from different training inputs and
strategies by the PDMI framework. PRN-MR-DE and RN-MR-DE denote ResNet trained
with and without a physics constraint using MRI and DECT images. PRN-DE and
RN-DE represent ResNet trained with and without a physics constraint using
DECT-only images. For the tissue surrogate study, PRN-MR-DE, PRN-DE, and
RN-MR-DE result in mean mass density errors: -0.72%, 2.62%, -3.58% for adipose;
-0.03%, -0.61%, and -0.18% for muscle; -0.58%, -1.36%, and -4.86% for 45% HA
bone. The retrospective patient study indicated that PRN-MR-DE predicted the
densities of soft tissue and bone within expected intervals based on the
literature survey, while PRN-DE generated large density deviations. The
proposed PDMI framework can generate accurate mass density maps using MRI and
DECT images. The physics-constrained training can further enhance model
efficacy, making PRN-MR-DE outperform RN-MR-DE. The patient investigation also
shows that the PDMI framework has the potential to improve proton range
uncertainty with accurate patient mass density maps.
