  Evaluating neural network performance is critical to deep neural network
design but a costly procedure. Neural predictors provide an efficient solution
by treating architectures as samples and learning to estimate their performance
on a given task. However, existing predictors are task-dependent, predominantly
estimating neural network performance on image classification benchmarks. They
are also search-space dependent; each predictor is designed to make predictions
for a specific architecture search space with predefined topologies and set of
operations. In this paper, we propose a novel All-in-One Predictor (AIO-P),
which aims to pretrain neural predictors on architecture examples from
multiple, separate computer vision (CV) task domains and multiple architecture
spaces, and then transfer to unseen downstream CV tasks or neural
architectures. We describe our proposed techniques for general graph
representation, efficient predictor pretraining and knowledge infusion
techniques, as well as methods to transfer to downstream tasks/spaces.
Extensive experimental results show that AIO-P can achieve Mean Absolute Error
(MAE) and Spearman's Rank Correlation (SRCC) below 1% and above 0.5,
respectively, on a breadth of target downstream CV tasks with or without
fine-tuning, outperforming a number of baselines. Moreover, AIO-P can directly
transfer to new architectures not seen during training, accurately rank them
and serve as an effective performance estimator when paired with an algorithm
designed to preserve performance while reducing FLOPs.
