  Conservation laws in a quantum many-body system play a direct role in its
dynamic behavior. Understanding the effect of weakly breaking a conservation
law due to coherent and incoherent errors is thus crucial, e.g., in the
realization of reliable quantum simulators. In this work, we perform exact
numerics and time-dependent perturbation theory to study the dynamics of
\textit{symmetry violation} in quantum many-body systems with slight coherent
(at strength $\lambda$) or incoherent (at strength $\gamma$) breaking of their
local and global symmetries. We rigorously prove the symmetry violation to be a
divergence measure in Hilbert space. Based on this, we show that symmetry
breaking generically leads to a crossover in the divergence growth from
diffusive behavior at onset times to ballistic or hyperballistic scaling at
intermediate times, before diffusion dominates at long times. More precisely,
we show that for local errors the leading coherent contribution to the symmetry
violation cannot be of order lower than $\propto\lambda t^2$ while its
leading-order incoherent counterpart is typically of order $\propto\gamma t$.
This remarkable interplay between unitary and incoherent gauge-breaking
scalings is also observed at higher orders in projectors onto symmetry
(super)sectors. Due to its occurrence at short times, the
diffusive-to-ballistic crossover is expected to be readily accessible in modern
ultracold-atom and NISQ-device experiments.
