  Existing Neural Architecture Search (NAS) methods either encode neural
architectures using discrete encodings that do not scale well, or adopt
supervised learning-based methods to jointly learn architecture representations
and optimize architecture search on such representations which incurs search
bias. Despite the widespread use, architecture representations learned in NAS
are still poorly understood. We observe that the structural properties of
neural architectures are hard to preserve in the latent space if architecture
representation learning and search are coupled, resulting in less effective
search performance. In this work, we find empirically that pre-training
architecture representations using only neural architectures without their
accuracies as labels considerably improve the downstream architecture search
efficiency. To explain these observations, we visualize how unsupervised
architecture representation learning better encourages neural architectures
with similar connections and operators to cluster together. This helps to map
neural architectures with similar performance to the same regions in the latent
space and makes the transition of architectures in the latent space relatively
smooth, which considerably benefits diverse downstream search strategies.
