  We inspect the long-term learning ability of Long Short-Term Memory language
models (LSTM LMs) by evaluating a contextual extension based on the Continuous
Bag-of-Words (CBOW) model for both sentence- and discourse-level LSTM LMs and
by analyzing its performance. We evaluate on text and speech. Sentence-level
models using the long-term contextual module perform comparably to vanilla
discourse-level LSTM LMs. On the other hand, the extension does not provide
gains for discourse-level models. These findings indicate that discourse-level
LSTM LMs already rely on contextual information to perform long-term learning.
