  Medical imaging datasets are inherently high dimensional with large
variability and low sample sizes that limit the effectiveness of deep learning
algorithms. Recently, generative adversarial networks (GANs) with the ability
to synthesize realist images have shown great potential as an alternative to
standard data augmentation techniques. Our work focuses on cross-modality
synthesis of fluorodeoxyglucose~(FDG) Positron Emission Tomography~(PET) scans
from structural Magnetic Resonance~(MR) images using generative models to
facilitate multi-modal diagnosis of Alzheimer's disease (AD). Specifically, we
propose a novel end-to-end, globally and locally aware image-to-image
translation GAN (GLA-GAN) with a multi-path architecture that enforces both
global structural integrity and fidelity to local details. We further
supplement the standard adversarial loss with voxel-level intensity,
multi-scale structural similarity (MS-SSIM) and region-of-interest (ROI) based
loss components that reduce reconstruction error, enforce structural
consistency at different scales and perceive variation in regional sensitivity
to AD respectively. Experimental results demonstrate that our GLA-GAN not only
generates synthesized FDG-PET scans with enhanced image quality but also
superior clinical utility in improving AD diagnosis compared to
state-of-the-art models. Finally, we attempt to interpret some of the internal
units of the GAN that are closely related to this specific cross-modality
generation task.
