  Recurrent Neural Network (RNN) has been widely used to tackle a wide variety
of language generation problems and are capable of attaining state-of-the-art
(SOTA) performance. However despite its impressive results, the large number of
parameters in the RNN model makes deployment to mobile and embedded devices
infeasible. Driven by this problem, many works have proposed a number of
pruning methods to reduce the sizes of the RNN model. In this work, we propose
an end-to-end pruning method for image captioning models equipped with visual
attention. Our proposed method is able to achieve sparsity levels up to 97.5%
without significant performance loss relative to the baseline (~ 2% loss at 40x
compression after fine-tuning). Our method is also simple to use and tune,
facilitating faster development times for neural network practitioners. We
perform extensive experiments on the popular MS-COCO dataset in order to
empirically validate the efficacy of our proposed method.
