  Symmetry is one of the most central concepts in physics, and it is no
surprise that it has also been widely adopted as an inductive bias for
machine-learning models applied to the physical sciences. This is especially
true for models targeting the properties of matter at the atomic scale. Both
established and state-of-the-art approaches, with almost no exceptions, are
built to be exactly equivariant to translations, permutations, and rotations of
the atoms. Incorporating symmetries -- rotations in particular -- constrains
the model design space and implies more complicated architectures that are
often also computationally demanding. There are indications that non-symmetric
models can easily learn symmetries from data, and that doing so can even be
beneficial for the accuracy of the model. We put a model that obeys rotational
invariance only approximately to the test, in realistic scenarios involving
simulations of gas-phase, liquid, and solid water. We focus specifically on
physical observables that are likely to be affected -- directly or indirectly
-- by symmetry breaking, finding negligible consequences when the model is used
in an interpolative, bulk, regime. Even for extrapolative gas-phase
predictions, the model remains very stable, even though symmetry artifacts are
noticeable. We also discuss strategies that can be used to systematically
reduce the magnitude of symmetry breaking when it occurs, and assess their
impact on the convergence of observables.
