  Medical imaging is limited by acquisition time and scanning equipment. CT and
MR volumes, reconstructed with thicker slices, are anisotropic with high
in-plane resolution and low through-plane resolution. We reveal an intriguing
phenomenon that due to the mentioned nature of data, performing slice-wise
interpolation from the axial view can yield greater benefits than performing
super-resolution from other views. Based on this observation, we propose an
Inter-Intra-slice Interpolation Network (I$^3$Net), which fully explores
information from high in-plane resolution and compensates for low through-plane
resolution. The through-plane branch supplements the limited information
contained in low through-plane resolution from high in-plane resolution and
enables continual and diverse feature learning. In-plane branch transforms
features to the frequency domain and enforces an equal learning opportunity for
all frequency bands in a global context learning paradigm. We further propose a
cross-view block to take advantage of the information from all three views
online. Extensive experiments on two public datasets demonstrate the
effectiveness of I$^3$Net, and noticeably outperforms state-of-the-art
super-resolution, video frame interpolation and slice interpolation methods by
a large margin. We achieve 43.90dB in PSNR, with at least 1.14dB improvement
under the upscale factor of $\times$2 on MSD dataset with faster inference.
Code is available at
https://github.com/DeepMed-Lab-ECNU/Medical-Image-Reconstruction.
