  The proliferation of pretrained models, as a result of advancements in
pretraining techniques, has led to the emergence of a vast zoo of publicly
available models. Effectively utilizing these resources to obtain models with
robust out-of-distribution generalization capabilities for downstream tasks has
become a crucial area of research. Previous research has primarily focused on
identifying the most powerful models within the model zoo, neglecting to fully
leverage the diverse inductive biases contained within. This paper argues that
the knowledge contained in weaker models is valuable and presents a method for
leveraging the diversity within the model zoo to improve out-of-distribution
generalization capabilities. Specifically, we investigate the behaviors of
various pretrained models across different domains of downstream tasks by
characterizing the variations in their encoded representations in terms of two
dimensions: diversity shift and correlation shift. This characterization
enables us to propose a new algorithm for integrating diverse pretrained
models, not limited to the strongest models, in order to achieve enhanced
out-of-distribution generalization performance. Our proposed method
demonstrates state-of-the-art empirical results on a variety of datasets, thus
validating the benefits of utilizing diverse knowledge.
