  Language models based on deep neural networks and traditional stochastic
modelling have become both highly functional and effective in recent times. In
this work, a general survey into the two types of language modelling is
conducted. We investigate the effectiveness of the Hidden Markov Model (HMM),
and the Long Short-Term Memory Model (LSTM). We analyze the hidden state
structures common to both models, and present an analysis on structural
similarity of the hidden states, common to both HMM's and LSTM's. We compare
the LSTM's predictive accuracy and hidden state output with respect to the HMM
for a varying number of hidden states. In this work, we justify that the less
complex HMM can serve as an appropriate approximation of the LSTM model.
