  Thanks to the use of convolution and pooling layers, convolutional neural
networks were for a long time thought to be shift-invariant. However, recent
works have shown that the output of a CNN can change significantly with small
shifts in input: a problem caused by the presence of downsampling (stride)
layers. The existing solutions rely either on data augmentation or on
anti-aliasing, both of which have limitations and neither of which enables
perfect shift invariance. Additionally, the gains obtained from these methods
do not extend to image patterns not seen during training. To address these
challenges, we propose adaptive polyphase sampling (APS), a simple sub-sampling
scheme that allows convolutional neural networks to achieve 100% consistency in
classification performance under shifts, without any loss in accuracy. With
APS, the networks exhibit perfect consistency to shifts even before training,
making it the first approach that makes convolutional neural networks truly
shift-invariant.
