  In out-of-distribution (OOD) generalization tasks, fine-tuning pre-trained
models has become a prevalent strategy. Different from most prior work that has
focused on advancing learning algorithms, we systematically examined how
pre-trained model size, pre-training dataset size, and training strategies
impact generalization and uncertainty calibration on downstream tasks. We
evaluated 100 models across diverse pre-trained model sizes, \update{five}
pre-training datasets, and five data augmentations through extensive
experiments on four distribution shift datasets totaling over 120,000 GPU
hours. Our results demonstrate the significant impact of pre-trained model
selection, with optimal choices substantially improving OOD accuracy over
algorithm improvement alone. We find larger models and bigger pre-training data
improve OOD performance and calibration, in contrast to some prior studies that
found modern deep networks to calibrate worse than classical shallow models.
Our work underscores the overlooked importance of pre-trained model selection
for out-of-distribution generalization and calibration.
