  Domain Generalization (DG) measures a classifier's ability to generalize to
new distributions of data it was not trained on. Recent work has shown that a
hyperparameter-tuned Empirical Risk Minimization (ERM) training procedure, that
is simply minimizing the empirical risk on the source domains, can outperform
most existing DG methods. ERM has achieved such strong results while only
tuning hyper-parameters such as learning rate, weight decay, batch size, and
dropout. However there are additional hyperparameters which further limit
overfitting and catastrophic forgetting. We therefore focus on tuning
previously untuned hyper-parameters, including training amount, initialization,
and additional regularizers. We call the resulting stronger baseline ERM++.
ERM++ improves the performance of DG by over 5% compared to prior ERM baselines
on a standard benchmark of 5 datasets with a ResNet-50 and over 15% with a
ViT-B/16, and outperforms all SOTA methods on DomainBed with both
architectures. We also explore the relationship between DG performance and
similarity to pre-training data, and find that similarity to pre-training data
distributions is an important driver of performance, but that ERM++ with
stronger initializations can deliver strong performance even on dissimilar
datasets.Code is released at https://github.com/piotr-teterwak/erm_plusplus.
