  Predictor-based algorithms have achieved remarkable performance in the Neural
Architecture Search (NAS) tasks. However, these methods suffer from high
computation costs, as training the performance predictor usually requires
training and evaluating hundreds of architectures from scratch. Previous works
along this line mainly focus on reducing the number of architectures required
to fit the predictor. In this work, we tackle this challenge from a different
perspective - improve search efficiency by cutting down the computation budget
of architecture training. We propose NOn-uniform Successive Halving (NOSH), a
hierarchical scheduling algorithm that terminates the training of
underperforming architectures early to avoid wasting budget. To effectively
leverage the non-uniform supervision signals produced by NOSH, we formulate
predictor-based architecture search as learning to rank with pairwise
comparisons. The resulting method - RANK-NOSH, reduces the search budget by ~5x
while achieving competitive or even better performance than previous
state-of-the-art predictor-based methods on various spaces and datasets.
