  To search an optimal sub-network within a general deep neural network (DNN),
existing neural architecture search (NAS) methods typically rely on
handcrafting a search space beforehand. Such requirements make it challenging
to extend them onto general scenarios without significant human expertise and
manual intervention. To overcome the limitations, we propose Automated
Search-Space Generation Neural Architecture Search (ASGNAS), perhaps the first
automated system to train general DNNs that cover all candidate connections and
operations and produce high-performing sub-networks in the one shot manner.
Technologically, ASGNAS delivers three noticeable contributions to minimize
human efforts: (i) automated search space generation for general DNNs; (ii) a
Hierarchical Half-Space Projected Gradient (H2SPG) that leverages the hierarchy
and dependency within generated search space to ensure the network validity
during optimization, and reliably produces a solution with both high
performance and hierarchical group sparsity; and (iii) automated sub-network
construction upon the H2SPG solution. Numerically, we demonstrate the
effectiveness of ASGNAS on a variety of general DNNs, including RegNet,
StackedUnets, SuperResNet, and DARTS, over benchmark datasets such as CIFAR10,
Fashion-MNIST, ImageNet, STL-10 , and SVNH. The sub-networks computed by ASGNAS
achieve competitive even superior performance compared to the starting full
DNNs and other state-of-the-arts. The library will be released at
https://github.com/tianyic/only_train_once.
