  Convolutional neural networks lack shift equivariance due to the presence of
downsampling layers. In image classification, adaptive polyphase downsampling
(APS-D) was recently proposed to make CNNs perfectly shift invariant. However,
in networks used for image reconstruction tasks, it can not by itself restore
shift equivariance. We address this problem by proposing adaptive polyphase
upsampling (APS-U), a non-linear extension of conventional upsampling, which
allows CNNs with symmetric encoder-decoder architecture (for example U-Net) to
exhibit perfect shift equivariance. With MRI and CT reconstruction experiments,
we show that networks containing APS-D/U layers exhibit state of the art
equivariance performance without sacrificing on image reconstruction quality.
In addition, unlike prior methods like data augmentation and anti-aliasing, the
gains in equivariance obtained from APS-D/U also extend to images outside the
training distribution.
