  This paper addresses an important problem of ranking the pre-trained deep
neural networks and screening the most transferable ones for downstream tasks.
It is challenging because the ground-truth model ranking for each task can only
be generated by fine-tuning the pre-trained models on the target dataset, which
is brute-force and computationally expensive. Recent advanced methods proposed
several lightweight transferability metrics to predict the fine-tuning results.
However, these approaches only capture static representations but neglect the
fine-tuning dynamics. To this end, this paper proposes a new transferability
metric, called \textbf{S}elf-challenging \textbf{F}isher \textbf{D}iscriminant
\textbf{A}nalysis (\textbf{SFDA}), which has many appealing benefits that
existing works do not have. First, SFDA can embed the static features into a
Fisher space and refine them for better separability between classes. Second,
SFDA uses a self-challenging mechanism to encourage different pre-trained
models to differentiate on hard examples. Third, SFDA can easily select
multiple pre-trained models for the model ensemble. Extensive experiments on
$33$ pre-trained models of $11$ downstream tasks show that SFDA is efficient,
effective, and robust when measuring the transferability of pre-trained models.
For instance, compared with the state-of-the-art method NLEEP, SFDA
demonstrates an average of $59.1$\% gain while bringing $22.5$x speedup in
wall-clock time. The code will be available at
\url{https://github.com/TencentARC/SFDA}.
