  Many algorithms and observed phenomena in deep learning appear to be affected
by parameter symmetries -- transformations of neural network parameters that do
not change the underlying neural network function. These include linear mode
connectivity, model merging, Bayesian neural network inference, metanetworks,
and several other characteristics of optimization or loss-landscapes. However,
theoretical analysis of the relationship between parameter space symmetries and
these phenomena is difficult. In this work, we empirically investigate the
impact of neural parameter symmetries by introducing new neural network
architectures that have reduced parameter space symmetries. We develop two
methods, with some provable guarantees, of modifying standard neural networks
to reduce parameter space symmetries. With these new methods, we conduct a
comprehensive experimental study consisting of multiple tasks aimed at
assessing the effect of removing parameter symmetries. Our experiments reveal
several interesting observations on the empirical impact of parameter
symmetries; for instance, we observe linear mode connectivity between our
networks without alignment of weight spaces, and we find that our networks
allow for faster and more effective Bayesian neural network training.
