  Structured distributions, i.e. distributions over combinatorial spaces, are
commonly used to learn latent probabilistic representations from observed data.
However, scaling these models is bottlenecked by the high computational and
memory complexity with respect to the size of the latent representations.
Common models such as Hidden Markov Models (HMMs) and Probabilistic
Context-Free Grammars (PCFGs) require time and space quadratic and cubic in the
number of hidden states respectively. This work demonstrates a simple approach
to reduce the computational and memory complexity of a large class of
structured models. We show that by viewing the central inference step as a
matrix-vector product and using a low-rank constraint, we can trade off model
expressivity and speed via the rank. Experiments with neural parameterized
structured models for language modeling, polyphonic music modeling,
unsupervised grammar induction, and video modeling show that our approach
matches the accuracy of standard models at large state spaces while providing
practical speedups.
