  Averaging neural network parameters is an intuitive method for fusing the
knowledge of two independent models. It is most prominently used in federated
learning. If models are averaged at the end of training, this can only lead to
a good performing model if the loss surface of interest is very particular,
i.e., the loss in the midpoint between the two models needs to be sufficiently
low. This is impossible to guarantee for the non-convex losses of
state-of-the-art networks. For averaging models trained on vastly different
datasets, it was proposed to average only the parameters of particular layers
or combinations of layers, resulting in better performing models. To get a
better understanding of the effect of layer-wise averaging, we analyse the
performance of the models that result from averaging single layers, or groups
of layers. Based on our empirical and theoretical investigation, we introduce a
novel notion of the layer-wise linear connectivity, and show that deep networks
do not have layer-wise barriers between them.
