  Convolutional neural networks (CNNs) have been increasingly deployed to edge
devices. Hence, many efforts have been made towards efficient CNN inference in
resource-constrained platforms. This paper attempts to explore an orthogonal
direction: how to conduct more energy-efficient training of CNNs, so as to
enable on-device training. We strive to reduce the energy cost during training,
by dropping unnecessary computations from three complementary levels:
stochastic mini-batch dropping on the data level; selective layer update on the
model level; and sign prediction for low-cost, low-precision back-propagation,
on the algorithm level. Extensive simulations and ablation studies, with real
energy measurements from an FPGA board, confirm the superiority of our proposed
strategies and demonstrate remarkable energy savings for training. For example,
when training ResNet-74 on CIFAR-10, we achieve aggressive energy savings of
>90% and >60%, while incurring a top-1 accuracy loss of only about 2% and 1.2%,
respectively. When training ResNet-110 on CIFAR-100, an over 84% training
energy saving is achieved without degrading inference accuracy.
