  Empirical studies of the loss landscape of deep networks have revealed that
many local minima are connected through low-loss valleys. Yet, little is known
about the theoretical origin of such valleys. We present a general framework
for finding continuous symmetries in the parameter space, which carve out
low-loss valleys. Our framework uses equivariances of the activation functions
and can be applied to different layer architectures. To generalize this
framework to nonlinear neural networks, we introduce a novel set of nonlinear,
data-dependent symmetries. These symmetries can transform a trained model such
that it performs similarly on new samples, which allows ensemble building that
improves robustness under certain adversarial attacks. We then show that
conserved quantities associated with linear symmetries can be used to define
coordinates along low-loss valleys. The conserved quantities help reveal that
using common initialization methods, gradient flow only explores a small part
of the global minimum. By relating conserved quantities to convergence rate and
sharpness of the minimum, we provide insights on how initialization impacts
convergence and generalizability.
