  Neural Architecture Search (NAS) has become a de facto approach in the recent
trend of AutoML to design deep neural networks (DNNs). Efficient or
near-zero-cost NAS proxies are further proposed to address the demanding
computational issues of NAS, where each candidate architecture network only
requires one iteration of backpropagation. The values obtained from the proxies
are considered the predictions of architecture performance on downstream tasks.
However, two significant drawbacks hinder the extended usage of Efficient NAS
proxies. (1) Efficient proxies are not adaptive to various search spaces. (2)
Efficient proxies are not extensible to multi-modality downstream tasks. Based
on the observations, we design a Extensible proxy (Eproxy) that utilizes
self-supervised, few-shot training (i.e., 10 iterations of backpropagation)
which yields near-zero costs. The key component that makes Eproxy efficient is
an untrainable convolution layer termed barrier layer that add the
non-linearities to the optimization spaces so that the Eproxy can discriminate
the performance of architectures in the early stage. Furthermore, to make
Eproxy adaptive to different downstream tasks/search spaces, we propose a
Discrete Proxy Search (DPS) to find the optimized training settings for Eproxy
with only handful of benchmarked architectures on the target tasks. Our
extensive experiments confirm the effectiveness of both Eproxy and Eproxy+DPS.
Code is available at https://github.com/leeyeehoo/GenNAS-Zero.
