  Differentiable neural architecture search methods became popular in recent
years, mainly due to their low search costs and flexibility in designing the
search space. However, these methods suffer the difficulty in optimizing
network, so that the searched network is often unfriendly to hardware. This
paper deals with this problem by adding a differentiable latency loss term into
optimization, so that the search process can tradeoff between accuracy and
latency with a balancing coefficient. The core of latency prediction is to
encode each network architecture and feed it into a multi-layer regressor, with
the training data which can be easily collected from randomly sampling a number
of architectures and evaluating them on the hardware. We evaluate our approach
on NVIDIA Tesla-P100 GPUs. With 100K sampled architectures (requiring a few
hours), the latency prediction module arrives at a relative error of lower than
10%. Equipped with this module, the search method can reduce the latency by 20%
meanwhile preserving the accuracy. Our approach also enjoys the ability of
being transplanted to a wide range of hardware platforms with very few efforts,
or being used to optimizing other non-differentiable factors such as power
consumption.
