  Fine-tuning of large pre-trained image and language models on small
customized datasets has become increasingly popular for improved prediction and
efficient use of limited resources. Fine-tuning requires identification of best
models to transfer-learn from and quantifying transferability prevents
expensive re-training on all of the candidate models/tasks pairs. In this
paper, we show that the statistical problems with covariance estimation drive
the poor performance of H-score -- a common baseline for newer metrics -- and
propose shrinkage-based estimator. This results in up to 80% absolute gain in
H-score correlation performance, making it competitive with the
state-of-the-art LogME measure. Our shrinkage-based H-score is
$3\times$-10$\times$ faster to compute compared to LogME. Additionally, we look
into a less common setting of target (as opposed to source) task selection. We
demonstrate previously overlooked problems in such settings with different
number of labels, class-imbalance ratios etc. for some recent metrics e.g.,
NCE, LEEP that resulted in them being misrepresented as leading measures. We
propose a correction and recommend measuring correlation performance against
relative accuracy in such settings. We support our findings with ~164,000
(fine-tuning trials) experiments on both vision models and graph neural
networks.
