  Using symmetry as an inductive bias in deep learning has been proven to be a
principled approach for sample-efficient model design. However, the
relationship between symmetry and the imperative for equivariance in neural
networks is not always obvious. Here, we analyze a key limitation that arises
in equivariant functions: their incapacity to break symmetry at the level of
individual data samples. In response, we introduce a novel notion of 'relaxed
equivariance' that circumvents this limitation. We further demonstrate how to
incorporate this relaxation into equivariant multilayer perceptrons (E-MLPs),
offering an alternative to the noise-injection method. The relevance of
symmetry breaking is then discussed in various application domains: physics,
graph representation learning, combinatorial optimization and equivariant
decoding.
