  We present a complimentary objective for training recurrent neural networks
(RNN) with gating units that helps with regularization and interpretability of
the trained model. Attention-based RNN models have shown success in many
difficult sequence to sequence classification problems with long and short term
dependencies, however these models are prone to overfitting. In this paper, we
describe how to regularize these models through an L1 penalty on the activation
of the gating units, and show that this technique reduces overfitting on a
variety of tasks while also providing to us a human-interpretable visualization
of the inputs used by the network. These tasks include sentiment analysis,
paraphrase recognition, and question answering.
