  We investigate the optimization of neural networks on symmetric data, and
compare the strategy of constraining the architecture to be equivariant to that
of using data augmentation. Our analysis reveals that that the relative
geometry of the admissible and the equivariant layers, respectively, plays a
key role. Under natural assumptions on the data, network, loss, and group of
symmetries, we show that compatibility of the spaces of admissible layers and
equivariant layers, in the sense that the corresponding orthogonal projections
commute, implies that the sets of equivariant stationary points are identical
for the two strategies. If the linear layers of the network also are given a
unitary parametrization, the set of equivariant layers is even invariant under
the gradient flow for augmented models. Our analysis however also reveals that
even in the latter situation, stationary points may be unstable for augmented
training although they are stable for the manifestly equivariant models.
