  Multiple imaging modalities are often used for disease diagnosis, prediction,
or population-based analyses. However, not all modalities might be available
due to cost, different study designs, or changes in imaging technology. If the
differences between the types of imaging are small, data harmonization
approaches can be used; for larger changes, direct image synthesis approaches
have been explored. In this paper, we develop an approach based on multi-modal
metric learning to synthesize images of diverse modalities. We use metric
learning via multi-modal image retrieval, resulting in embeddings that can
relate images of different modalities. Given a large image database, the
learned image embeddings allow us to use k-nearest neighbor (k-NN) regression
for image synthesis. Our driving medical problem is knee osteoarthritis (KOA),
but our developed method is general after proper image alignment. We test our
approach by synthesizing cartilage thickness maps obtained from 3D magnetic
resonance (MR) images using 2D radiographs. Our experiments show that the
proposed method outperforms direct image synthesis and that the synthesized
thickness maps retain information relevant to downstream tasks such as
progression prediction and Kellgren-Lawrence grading (KLG). Our results suggest
that retrieval approaches can be used to obtain high-quality and meaningful
image synthesis results given large image databases.
