  During the deployment of deep neural networks (DNNs) on edge devices, many
research efforts are devoted to the limited hardware resource. However, little
attention is paid to the influence of dynamic power management. As edge devices
typically only have a budget of energy with batteries (rather than almost
unlimited energy support on servers or workstations), their dynamic power
management often changes the execution frequency as in the widely-used dynamic
voltage and frequency scaling (DVFS) technique. This leads to highly unstable
inference speed performance, especially for computation-intensive DNN models,
which can harm user experience and waste hardware resources. We firstly
identify this problem and then propose All-in-One, a highly representative
pruning framework to work with dynamic power management using DVFS. The
framework can use only one set of model weights and soft masks (together with
other auxiliary parameters of negligible storage) to represent multiple models
of various pruning ratios. By re-configuring the model to the corresponding
pruning ratio for a specific execution frequency (and voltage), we are able to
achieve stable inference speed, i.e., keeping the difference in speed
performance under various execution frequencies as small as possible. Our
experiments demonstrate that our method not only achieves high accuracy for
multiple models of different pruning ratios, but also reduces their variance of
inference latency for various frequencies, with minimal memory consumption of
only one model and one soft mask.
