  The use of neural network parametrizations to represent the ground state in
variational Monte Carlo (VMC) calculations has generated intense interest in
recent years. However, as we demonstrate in the context of the periodic
Heisenberg spin chain, this approach can produce unreliable wave function
approximations. One of the most obvious signs of failure is the occurrence of
random, persistent spikes in the energy estimate during training. These energy
spikes are caused by regions of configuration space that are over-represented
by the wave function density, which are called ``spurious modes'' in the
machine learning literature. After exploring these spurious modes in detail, we
demonstrate that a collective-variable-based penalization yields a
substantially more robust training procedure, preventing the formation of
spurious modes and improving the accuracy of energy estimates. Because the
penalization scheme is cheap to implement and is not specific to the particular
model studied here, it can be extended to other applications of VMC where a
reasonable choice of collective variable is available.
