  This paper strives to generate a synthetic computed tomography (CT) image
from a magnetic resonance (MR) image. The synthetic CT image is valuable for
radiotherapy planning when only an MR image is available. Recent approaches
have made large strides in solving this challenging synthesis problem with
convolutional neural networks that learn a mapping from MR inputs to CT
outputs. In this paper, we find that all existing approaches share a common
limitation: reconstruction breaks down in and around the high-frequency parts
of CT images. To address this common limitation, we introduce
frequency-supervised deep networks to explicitly enhance high-frequency
MR-to-CT image reconstruction. We propose a frequency decomposition layer that
learns to decompose predicted CT outputs into low- and high-frequency
components, and we introduce a refinement module to improve high-frequency
reconstruction through high-frequency adversarial learning. Experimental
results on a new dataset with 45 pairs of 3D MR-CT brain images show the
effectiveness and potential of the proposed approach. Code is available at
\url{https://github.com/shizenglin/Frequency-Supervised-MR-to-CT-Image-Synthesis}.
