  Differentiable Architecture Search (DARTS) is a simple yet efficient Neural
Architecture Search (NAS) method. During the search stage, DARTS trains a
supernet by jointly optimizing architecture parameters and network parameters.
During the evaluation stage, DARTS discretizes the supernet to derive the
optimal architecture based on architecture parameters. However, recent research
has shown that during the training process, the supernet tends to converge
towards sharp minima rather than flat minima. This is evidenced by the higher
sharpness of the loss landscape of the supernet, which ultimately leads to a
performance gap between the supernet and the optimal architecture. In this
paper, we propose Self-Distillation Differentiable Neural Architecture Search
(SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to
distill knowledge from previous steps of the supernet to guide its training in
the current step, effectively reducing the sharpness of the supernet's loss and
bridging the performance gap between the supernet and the optimal architecture.
Furthermore, we introduce the concept of voting teachers, where multiple
previous supernets are selected as teachers, and their output probabilities are
aggregated through voting to obtain the final teacher prediction. Experimental
results on real datasets demonstrate the advantages of our novel
self-distillation-based NAS method compared to state-of-the-art alternatives.
