  Differentiable neural architecture search (DARTS) is a popular method for
neural architecture search (NAS), which performs cell-search and utilizes
continuous relaxation to improve the search efficiency via gradient-based
optimization. The main shortcoming of DARTS is performance collapse, where the
discovered architecture suffers from a pattern of declining quality during
search. Performance collapse has become an important topic of research, with
many methods trying to solve the issue through either regularization or
fundamental changes to DARTS. However, the weight-sharing framework used for
cell-search in DARTS and the convergence of architecture parameters has not
been analyzed yet. In this paper, we provide a thorough and novel theoretical
and empirical analysis on DARTS and its point of convergence. We show that
DARTS suffers from a specific structural flaw due to its weight-sharing
framework that limits the convergence of DARTS to saturation points of the
softmax function. This point of convergence gives an unfair advantage to layers
closer to the output in choosing the optimal architecture, causing performance
collapse. We then propose two new regularization terms that aim to prevent
performance collapse by harmonizing operation selection via aligning gradients
of layers. Experimental results on six different search spaces and three
different datasets show that our method ($\Lambda$-DARTS) does indeed prevent
performance collapse, providing justification for our theoretical analysis and
the proposed remedy.
