  Autoregressive state transitions, where predictions are conditioned on past
predictions, are the predominant choice for both deterministic and stochastic
sequential models. However, autoregressive feedback exposes the evolution of
the hidden state trajectory to potential biases from well-known train-test
discrepancies. In this paper, we combine a latent state space model with a CRF
observation model. We argue that such autoregressive observation models form an
interesting middle ground that expresses local correlations on the word level
but keeps the state evolution non-autoregressive. On unconditional sentence
generation we show performance improvements compared to RNN and GAN baselines
while avoiding some prototypical failure modes of autoregressive models.
