  One-shot neural architecture search (NAS) substantially improves the search
efficiency by training one supernet to estimate the performance of every
possible child architecture (i.e., subnet). However, the inconsistency of
characteristics among subnets incurs serious interference in the optimization,
resulting in poor performance ranking correlation of subnets. Subsequent
explorations decompose supernet weights via a particular criterion, e.g.,
gradient matching, to reduce the interference; yet they suffer from huge
computational cost and low space separability. In this work, we propose a
lightweight and effective local intrinsic dimension (LID)-based method NAS-LID.
NAS-LID evaluates the geometrical properties of architectures by calculating
the low-cost LID features layer-by-layer, and the similarity characterized by
LID enjoys better separability compared with gradients, which thus effectively
reduces the interference among subnets. Extensive experiments on NASBench-201
indicate that NAS-LID achieves superior performance with better efficiency.
Specifically, compared to the gradient-driven method, NAS-LID can save up to
86% of GPU memory overhead when searching on NASBench-201. We also demonstrate
the effectiveness of NAS-LID on ProxylessNAS and OFA spaces. Source code:
https://github.com/marsggbo/NAS-LID.
