  While selecting the hyper-parameters of Neural Networks (NNs) has been so far
treated as an art, the emergence of more complex, deeper architectures poses
increasingly more challenges to designers and Machine Learning (ML)
practitioners, especially when power and memory constraints need to be
considered. In this work, we propose HyperPower, a framework that enables
efficient Bayesian optimization and random search in the context of power- and
memory-constrained hyper-parameter optimization for NNs running on a given
hardware platform. HyperPower is the first work (i) to show that power
consumption can be used as a low-cost, a priori known constraint, and (ii) to
propose predictive models for the power and memory of NNs executing on GPUs.
Thanks to HyperPower, the number of function evaluations and the best test
error achieved by a constraint-unaware method are reached up to 112.99x and
30.12x faster, respectively, while never considering invalid configurations.
HyperPower significantly speeds up the hyper-parameter optimization, achieving
up to 57.20x more function evaluations compared to constraint-unaware methods
for a given time interval, effectively yielding significant accuracy
improvements by up to 67.6%.
