  LSTMs have become a basic building block for many deep NLP models. In recent
years, many improvements and variations have been proposed for deep sequence
models in general, and LSTMs in particular. We propose and analyze a series of
augmentations and modifications to LSTM networks resulting in improved
performance for text classification datasets. We observe compounding
improvements on traditional LSTMs using Monte Carlo test-time model averaging,
average pooling, and residual connections, along with four other suggested
modifications. Our analysis provides a simple, reliable, and high quality
baseline model.
