  Figuring out which Pre-Trained Model (PTM) from a model zoo fits the target
task is essential to take advantage of plentiful model resources. With the
availability of numerous heterogeneous PTMs from diverse fields, efficiently
selecting the most suitable PTM is challenging due to the time-consuming costs
of carrying out forward or backward passes over all PTMs. In this paper, we
propose Model Spider, which tokenizes both PTMs and tasks by summarizing their
characteristics into vectors to enable efficient PTM selection. By leveraging
the approximated performance of PTMs on a separate set of training tasks, Model
Spider learns to construct tokens and measure the fitness score between a
model-task pair via their tokens. The ability to rank relevant PTMs higher than
others generalizes to new tasks. With the top-ranked PTM candidates, we further
learn to enrich task tokens with their PTM-specific semantics to re-rank the
PTMs for better selection. Model Spider balances efficiency and selection
ability, making PTM selection like a spider preying on a web. Model Spider
demonstrates promising performance in various configurations of model zoos.
