  Zero-cost proxies (ZC proxies) are a recent architecture performance
prediction technique aiming to significantly speed up algorithms for neural
architecture search (NAS). Recent work has shown that these techniques show
great promise, but certain aspects, such as evaluating and exploiting their
complementary strengths, are under-studied. In this work, we create
NAS-Bench-Suite: we evaluate 13 ZC proxies across 28 tasks, creating by far the
largest dataset (and unified codebase) for ZC proxies, enabling
orders-of-magnitude faster experiments on ZC proxies, while avoiding
confounding factors stemming from different implementations. To demonstrate the
usefulness of NAS-Bench-Suite, we run a large-scale analysis of ZC proxies,
including a bias analysis, and the first information-theoretic analysis which
concludes that ZC proxies capture substantial complementary information.
Motivated by these findings, we present a procedure to improve the performance
of ZC proxies by reducing biases such as cell size, and we also show that
incorporating all 13 ZC proxies into the surrogate models used by NAS
algorithms can improve their predictive performance by up to 42%. Our code and
datasets are available at https://github.com/automl/naslib/tree/zerocost.
