  We address the problem that state-of-the-art Convolution Neural Networks
(CNN) classifiers are not invariant to small shifts. The problem can be solved
by the removal of sub-sampling operations such as stride and max pooling, but
at a cost of severely degraded training and test efficiency. We present a novel
usage of Gaussian-Hermite basis to efficiently approximate arbitrary filters
within the CNN framework to obtain translation invariance. This is shown to be
invariant to small shifts, and preserves the efficiency of training. Further,
to improve efficiency in memory usage as well as computational speed, we show
that it is still possible to sub-sample with this approach and retain a weaker
form of invariance that we call \emph{translation insensitivity}, which leads
to stability with respect to shifts. We prove these claims analytically and
empirically. Our analytic methods further provide a framework for understanding
any architecture in terms of translation insensitivity, and provide guiding
principles for design.
