  Over the past years, great progress has been made in improving the computing
power of general-purpose graphics processing units (GPGPUs), which facilitates
the prosperity of deep neural networks (DNNs) in multiple fields like computer
vision and natural language processing. A typical DNN training process
repeatedly updates tens of millions of parameters, which not only requires huge
computing resources but also consumes significant energy. In order to train
DNNs in a more energy-efficient way, we empirically investigate the impact of
GPU Dynamic Voltage and Frequency Scaling (DVFS) on the energy consumption and
performance of deep learning. Our experiments cover a wide range of GPU
architectures, DVFS settings, and DNN configurations. We observe that, compared
to the default core frequency settings of three tested GPUs, the optimal core
frequency can help conserve 8.7%$\sim$23.1% energy consumption for different
DNN training cases. Regarding the inference, the benefits vary from
19.6%$\sim$26.4%. Our findings suggest that GPU DVFS has great potentials to
help develop energy efficient DNN training/inference schemes.
