  Magnetic Resonance Imaging (MRI) offers high-resolution \emph{in vivo}
imaging and rich functional and anatomical multimodality tissue contrast. In
practice, however, there are challenges associated with considerations of
scanning costs, patient comfort, and scanning time that constrain how much data
can be acquired in clinical or research studies. In this paper, we explore the
possibility of generating high-resolution and multimodal images from
low-resolution single-modality imagery. We propose the weakly-supervised joint
convolutional sparse coding to simultaneously solve the problems of
super-resolution (SR) and cross-modality image synthesis. The learning process
requires only a few registered multimodal image pairs as the training set.
Additionally, the quality of the joint dictionary learning can be improved
using a larger set of unpaired images. To combine unpaired data from different
image resolutions/modalities, a hetero-domain image alignment term is proposed.
Local image neighborhoods are naturally preserved by operating on the whole
image domain (as opposed to image patches) and using joint convolutional sparse
coding. The paired images are enhanced in the joint learning process with
unpaired data and an additional maximum mean discrepancy term, which minimizes
the dissimilarity between their feature distributions. Experiments show that
the proposed method outperforms state-of-the-art techniques on both SR
reconstruction and simultaneous SR and cross-modality synthesis.
