  We propose neural network layers that explicitly combine frequency and image
feature representations and show that they can be used as a versatile building
block for reconstruction from frequency space data. Our work is motivated by
the challenges arising in MRI acquisition where the signal is a corrupted
Fourier transform of the desired image. The proposed joint learning schemes
enable both correction of artifacts native to the frequency space and
manipulation of image space representations to reconstruct coherent image
structures at every layer of the network. This is in contrast to most current
deep learning approaches for image reconstruction that treat frequency and
image space features separately and often operate exclusively in one of the two
spaces. We demonstrate the advantages of joint convolutional learning for a
variety of tasks, including motion correction, denoising, reconstruction from
undersampled acquisitions, and combined undersampling and motion correction on
simulated and real world multicoil MRI data. The joint models produce
consistently high quality output images across all tasks and datasets. When
integrated into a state of the art unrolled optimization network with
physics-inspired data consistency constraints for undersampled reconstruction,
the proposed architectures significantly improve the optimization landscape,
which yields an order of magnitude reduction of training time. This result
suggests that joint representations are particularly well suited for MRI
signals in deep learning networks. Our code and pretrained models are publicly
available at https://github.com/nalinimsingh/interlacer.
