  In this paper we look into the conjecture of Entezari et al. (2021) which
states that if the permutation invariance of neural networks is taken into
account, then there is likely no loss barrier to the linear interpolation
between SGD solutions. First, we observe that neuron alignment methods alone
are insufficient to establish low-barrier linear connectivity between SGD
solutions due to a phenomenon we call variance collapse: interpolated deep
networks suffer a collapse in the variance of their activations, causing poor
performance. Next, we propose REPAIR (REnormalizing Permuted Activations for
Interpolation Repair) which mitigates variance collapse by rescaling the
preactivations of such interpolated networks. We explore the interaction
between our method and the choice of normalization layer, network width, and
depth, and demonstrate that using REPAIR on top of neuron alignment methods
leads to 60%-100% relative barrier reduction across a wide variety of
architecture families and tasks. In particular, we report a 74% barrier
reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on
CIFAR10.
