  With the preponderance of pretrained deep learning models available
off-the-shelf from model banks today, finding the best weights to fine-tune to
your use-case can be a daunting task. Several methods have recently been
proposed to find good models for transfer learning, but they either don't scale
well to large model banks or don't perform well on the diversity of
off-the-shelf models. Ideally the question we want to answer is, "given some
data and a source model, can you quickly predict the model's accuracy after
fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse
Model Selection" and propose several benchmarks for evaluating on this task. We
find that existing model selection and transferability estimation methods
perform poorly here and analyze why this is the case. We then introduce simple
techniques to improve the performance and speed of these algorithms. Finally,
we iterate on existing methods to create PARC, which outperforms all other
methods on diverse model selection. We have released the benchmarks and method
code in hope to inspire future work in model selection for accessible transfer
learning.
