  This work investigates the computational expressivity of language models
(LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992)
famously showed that RNNs with rational weights and hidden states and unbounded
computation time are Turing complete. However, LMs define weightings over
strings in addition to just (unweighted) language membership and the analysis
of the computational power of RNN LMs (RLMs) should reflect this. We extend the
Turing completeness result to the probabilistic case, showing how a rationally
weighted RLM with unbounded computation time can simulate any deterministic
probabilistic Turing machine (PTM) with rationally weighted transitions. Since,
in practice, RLMs work in real-time, processing a symbol at every time step, we
treat the above result as an upper bound on the expressivity of RLMs. We also
provide a lower bound by showing that under the restriction to real-time
computation, such models can simulate deterministic real-time rational PTMs.
