  State-of-the-art convolutional neural networks (CNNs) used in vision
applications have large models with numerous weights. Training these models is
very compute- and memory-resource intensive. Much research has been done on
pruning or compressing these models to reduce the cost of inference, but little
work has addressed the costs of training. We focus precisely on accelerating
training. We propose PruneTrain, a cost-efficient mechanism that gradually
reduces the training cost during training. PruneTrain uses a structured
group-lasso regularization approach that drives the training optimization
toward both high accuracy and small weight values. Small weights can then be
periodically removed by reconfiguring the network model to a smaller one. By
using a structured-pruning approach and additional reconfiguration techniques
we introduce, the pruned model can still be efficiently processed on a GPU
accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end
training time of ResNet50 for ImageNet by reducing computation cost by 40% in
FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the
inter-accelerator communication by 55%.
