  Medical imaging is a cornerstone of therapy and diagnosis in modern medicine.
However, the choice of imaging modality for a particular theranostic task
typically involves trade-offs between the feasibility of using a particular
modality (e.g., short wait times, low cost, fast acquisition, reduced
radiation/invasiveness) and the expected performance on a clinical task (e.g.,
diagnostic accuracy, efficacy of treatment planning and guidance). In this
work, we aim to apply the knowledge learned from the less feasible but
better-performing (superior) modality to guide the utilization of the
more-feasible yet under-performing (inferior) modality and steer it towards
improved performance. We focus on the application of deep learning for
image-based diagnosis. We develop a light-weight guidance model that leverages
the latent representation learned from the superior modality, when training a
model that consumes only the inferior modality. We examine the advantages of
our method in the context of two clinical applications: multi-task skin lesion
classification from clinical and dermoscopic images and brain tumor
classification from multi-sequence magnetic resonance imaging (MRI) and
histopathology images. For both these scenarios we show a boost in diagnostic
performance of the inferior modality without requiring the superior modality.
Furthermore, in the case of brain tumor classification, our method outperforms
the model trained on the superior modality while producing comparable results
to the model that uses both modalities during inference.
