  Standard neural networks struggle to generalize under distribution shifts in
computer vision. Fortunately, combining multiple networks can consistently
improve out-of-distribution generalization. In particular, weight averaging
(WA) strategies were shown to perform best on the competitive DomainBed
benchmark; they directly average the weights of multiple networks despite their
nonlinearities. In this paper, we propose Diverse Weight Averaging (DiWA), a
new WA strategy whose main motivation is to increase the functional diversity
across averaged models. To this end, DiWA averages weights obtained from
several independent training runs: indeed, models obtained from different runs
are more diverse than those collected along a single run thanks to differences
in hyperparameters and training procedures. We motivate the need for diversity
by a new bias-variance-covariance-locality decomposition of the expected error,
exploiting similarities between WA and standard functional ensembling.
Moreover, this decomposition highlights that WA succeeds when the variance term
dominates, which we show occurs when the marginal distribution changes at test
time. Experimentally, DiWA consistently improves the state of the art on
DomainBed without inference overhead.
