  Differentiable architecture search (DARTS) has been a popular one-shot
paradigm for NAS due to its high efficiency. It introduces trainable
architecture parameters to represent the importance of candidate operations and
proposes first/second-order approximation to estimate their gradients, making
it possible to solve NAS by gradient descent algorithm. However, our in-depth
empirical results show that the approximation will often distort the loss
landscape, leading to the biased objective to optimize and in turn inaccurate
gradient estimation for architecture parameters. This work turns to zero-order
optimization and proposes a novel NAS scheme, called ZARTS, to search without
enforcing the above approximation. Specifically, three representative
zero-order optimization methods are introduced: RS, MGS, and GLD, among which
MGS performs best by balancing the accuracy and speed. Moreover, we explore the
connections between RS/MGS and gradient descent algorithm and show that our
ZARTS can be seen as a robust gradient-free counterpart to DARTS. Extensive
experiments on multiple datasets and search spaces show the remarkable
performance of our method. In particular, results on 12 benchmarks verify the
outstanding robustness of ZARTS, where the performance of DARTS collapses due
to its known instability issue. Also, we search on the search space of DARTS to
compare with peer methods, and our discovered architecture achieves 97.54%
accuracy on CIFAR-10 and 75.7% top-1 accuracy on ImageNet, which are
state-of-the-art performance.
