  With the increasing abundance of pretrained models in recent years, the
problem of selecting the best pretrained checkpoint for a particular downstream
classification task has been gaining increased attention. Although several
methods have recently been proposed to tackle the selection problem (e.g. LEEP,
H-score), these methods resort to applying heuristics that are not well
motivated by learning theory. In this paper we present PACTran, a theoretically
grounded family of metrics for pretrained model selection and transferability
measurement. We first show how to derive PACTran metrics from the optimal
PAC-Bayesian bound under the transfer learning setting. We then empirically
evaluate three metric instantiations of PACTran on a number of vision tasks
(VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the
results shows PACTran is a more consistent and effective transferability
measure compared to existing selection methods.
