  Differentiable architecture search is prevalent in the field of NAS because
of its simplicity and efficiency, where two paradigms, multi-path algorithms
and single-path methods, are dominated. Multi-path framework (e.g. DARTS) is
intuitive but suffers from memory usage and training collapse. Single-path
methods (e.g.GDAS and ProxylessNAS) mitigate the memory issue and shrink the
gap between searching and evaluation but sacrifice the performance. In this
paper, we propose a conceptually simple yet efficient method to bridge these
two paradigms, referred as Mutually-aware Sub-Graphs Differentiable
Architecture Search (MSG-DAS). The core of our framework is a differentiable
Gumbel-TopK sampler that produces multiple mutually exclusive single-path
sub-graphs. To alleviate the severer skip-connect issue brought by multiple
sub-graphs setting, we propose a Dropblock-Identity module to stabilize the
optimization. To make best use of the available models (super-net and
sub-graphs), we introduce a memory-efficient super-net guidance distillation to
improve training. The proposed framework strikes a balance between flexible
memory usage and searching quality. We demonstrate the effectiveness of our
methods on ImageNet and CIFAR10, where the searched models show a comparable
performance as the most recent approaches.
