  Recent studies suggest that with sufficiently wide models, most SGD solutions
can, up to permutation, converge into the same basin. This phenomenon, known as
the model re-basin regime, has significant implications for model averaging by
ensuring the linear mode connectivity. However, current re-basin strategies are
ineffective in many scenarios due to a lack of comprehensive understanding of
underlying mechanisms. Addressing this gap, this paper provides novel insights
into understanding and improving the standard practice. Firstly, we decompose
re-normalization into rescaling and reshift, uncovering that rescaling plays a
crucial role in re-normalization while re-basin performance is sensitive to
shifts in model activation. The finding calls for a more nuanced handling of
the activation shift. Secondly, we identify that the merged model suffers from
the issue of activation collapse and magnitude collapse. Varying the learning
rate, weight decay, and initialization method can mitigate the issues and
improve model performance. Lastly, we propose a new perspective to unify the
re-basin and pruning, under which a lightweight yet effective post-pruning
technique is derived, which can significantly improve the model performance
after pruning. Our implementation is available at
https://github.com/XingyuQu/rethink-re-basin.
