  Neural Architecture Search (NAS) has emerged as a favoured method for
unearthing effective neural architectures. Recent development of large models
has intensified the demand for faster search speeds and more accurate search
results. However, designing large models by NAS is challenging due to the
dramatical increase of search space and the associated huge performance
evaluation cost. Consider a typical modular search space widely used in NAS, in
which a neural architecture consists of $m$ block nodes and a block node has
$n$ alternative blocks. Facing the space containing $n^m$ candidate networks,
existing NAS methods attempt to find the best one by searching and evaluating
candidate networks directly.Different from the general strategy that takes
architecture search as a whole problem, we propose a novel divide-and-conquer
strategy by making use of the modular nature of the search space.Here, we
introduce MathNAS, a general NAS framework based on mathematical programming.In
MathNAS, the performances of the $m*n$ possible building blocks in the search
space are calculated first, and then the performance of a network is directly
predicted based on the performances of its building blocks. Although estimating
block performances involves network training, just as what happens for network
performance evaluation in existing NAS methods, predicting network performance
is completely training-free and thus extremely fast. In contrast to the $n^m$
candidate networks to evaluate in existing NAS methods, which require training
and a formidable computational burden, there are only $m*n$ possible blocks to
handle in MathNAS. Therefore, our approach effectively reduces the complexity
of network performance evaluation.Our code is available at
https://github.com/wangqinsi1/MathNAS.
