  Low dimensional representations of words allow accurate NLP models to be
trained on limited annotated data. While most representations ignore words'
local context, a natural way to induce context-dependent representations is to
perform inference in a probabilistic latent-variable sequence model. Given the
recent success of continuous vector space word representations, we provide such
an inference procedure for continuous states, where words' representations are
given by the posterior mean of a linear dynamical system. Here, efficient
inference can be performed using Kalman filtering. Our learning algorithm is
extremely scalable, operating on simple cooccurrence counts for both parameter
initialization using the method of moments and subsequent iterations of EM. In
our experiments, we employ our inferred word embeddings as features in standard
tagging tasks, obtaining significant accuracy improvements. Finally, the Kalman
filter updates can be seen as a linear recurrent neural network. We demonstrate
that using the parameters of our model to initialize a non-linear recurrent
neural network language model reduces its training time by a day and yields
lower perplexity.
