  We propose a novel method to increase shift invariance and prediction
accuracy in convolutional neural networks. Specifically, we replace the
first-layer combination "real-valued convolutions + max pooling" (RMax) by
"complex-valued convolutions + modulus" (CMod), which is stable to
translations, or shifts. To justify our approach, we claim that CMod and RMax
produce comparable outputs when the convolution kernel is band-pass and
oriented (Gabor-like filter). In this context, CMod can therefore be considered
as a stable alternative to RMax. To enforce this property, we constrain the
convolution kernels to adopt such a Gabor-like structure. The corresponding
architecture is called mathematical twin, because it employs a well-defined
mathematical operator to mimic the behavior of the original, freely-trained
model. Our approach achieves superior accuracy on ImageNet and CIFAR-10
classification tasks, compared to prior methods based on low-pass filtering.
Arguably, our approach's emphasis on retaining high-frequency details
contributes to a better balance between shift invariance and information
preservation, resulting in improved performance. Furthermore, it has a lower
computational cost and memory footprint than concurrent work, making it a
promising solution for practical implementation.
