  Designing energy-efficient networks is of critical importance for enabling
state-of-the-art deep learning in mobile and edge settings where the
computation and energy budgets are highly limited. Recently, Liu et al. (2019)
framed the search of efficient neural architectures into a continuous splitting
process: it iteratively splits existing neurons into multiple off-springs to
achieve progressive loss minimization, thus finding novel architectures by
gradually growing the neural network. However, this method was not specifically
tailored for designing energy-efficient networks, and is computationally
expensive on large-scale benchmarks. In this work, we substantially improve Liu
et al. (2019) in two significant ways: 1) we incorporate the energy cost of
splitting different neurons to better guide the splitting process, thereby
discovering more energy-efficient network architectures; 2) we substantially
speed up the splitting process of Liu et al. (2019), which requires expensive
eigen-decomposition, by proposing a highly scalable Rayleigh-quotient
stochastic gradient algorithm. Our fast algorithm allows us to reduce the
computational cost of splitting to the same level of typical back-propagation
updates and enables efficient implementation on GPU. Extensive empirical
results show that our method can train highly accurate and energy-efficient
networks on challenging datasets such as ImageNet, improving a variety of
baselines, including the pruning-based methods and expert-designed
architectures.
