  One of the key steps in Neural Architecture Search (NAS) is to estimate the
performance of candidate architectures. Existing methods either directly use
the validation performance or learn a predictor to estimate the performance.
However, these methods can be either computationally expensive or very
inaccurate, which may severely affect the search efficiency and performance.
Moreover, as it is very difficult to annotate architectures with accurate
performance on specific tasks, learning a promising performance predictor is
often non-trivial due to the lack of labeled data. In this paper, we argue that
it may not be necessary to estimate the absolute performance for NAS. On the
contrary, we may need only to understand whether an architecture is better than
a baseline one. However, how to exploit this comparison information as the
reward and how to well use the limited labeled data remains two great
challenges. In this paper, we propose a novel Contrastive Neural Architecture
Search (CTNAS) method which performs architecture search by taking the
comparison results between architectures as the reward. Specifically, we design
and learn a Neural Architecture Comparator (NAC) to compute the probability of
candidate architectures being better than a baseline one. Moreover, we present
a baseline updating scheme to improve the baseline iteratively in a curriculum
learning manner. More critically, we theoretically show that learning NAC is
equivalent to optimizing the ranking over architectures. Extensive experiments
in three search spaces demonstrate the superiority of our CTNAS over existing
methods.
