  This survey focuses on categorizing and evaluating the methods of supernet
optimization in the field of Neural Architecture Search (NAS). Supernet
optimization involves training a single, over-parameterized network that
encompasses the search space of all possible network architectures. The survey
analyses supernet optimization methods based on their approaches to spatial and
temporal optimization. Spatial optimization relates to optimizing the
architecture and parameters of the supernet and its subnets, while temporal
optimization deals with improving the efficiency of selecting architectures
from the supernet. The benefits, limitations, and potential applications of
these methods in various tasks and settings, including transferability, domain
generalization, and Transformer models, are also discussed.
