  Convolutional neural networks have been extremely successful in the image
recognition domain because they ensure equivariance to translations. There have
been many recent attempts to generalize this framework to other domains,
including graphs and data lying on manifolds. In this paper we give a rigorous,
theoretical treatment of convolution and equivariance in neural networks with
respect to not just translations, but the action of any compact group. Our main
result is to prove that (given some natural constraints) convolutional
structure is not just a sufficient, but also a necessary condition for
equivariance to the action of a compact group. Our exposition makes use of
concepts from representation theory and noncommutative harmonic analysis and
derives new generalized convolution formulae.
