  In neural architecture search (NAS) methods based on latent space
optimization (LSO), a deep generative model is trained to embed discrete neural
architectures into a continuous latent space. In this case, different
optimization algorithms that operate in the continuous space can be implemented
to search neural architectures. However, the optimization of latent variables
is challenging for gradient-based LSO since the mapping from the latent space
to the architecture performance is generally non-convex. To tackle this
problem, this paper develops a convexity regularized latent space optimization
(CR-LSO) method, which aims to regularize the learning process of latent space
in order to obtain a convex architecture performance mapping. Specifically,
CR-LSO trains a graph variational autoencoder (G-VAE) to learn the continuous
representations of discrete architectures. Simultaneously, the learning process
of latent space is regularized by the guaranteed convexity of input convex
neural networks (ICNNs). In this way, the G-VAE is forced to learn a convex
mapping from the architecture representation to the architecture performance.
Hereafter, the CR-LSO approximates the performance mapping using the ICNN and
leverages the estimated gradient to optimize neural architecture
representations. Experimental results on three popular NAS benchmarks show that
CR-LSO achieves competitive evaluation results in terms of both computational
complexity and architecture performance.
