  Autoregressive feedback is considered a necessity for successful
unconditional text generation using stochastic sequence models. However, such
feedback is known to introduce systematic biases into the training process and
it obscures a principle of generation: committing to global information and
forgetting local nuances. We show that a non-autoregressive deep state space
model with a clear separation of global and local uncertainty can be built from
only two ingredients: An independent noise source and a deterministic
transition function. Recent advances on flow-based variational inference can be
used to train an evidence lower-bound without resorting to annealing, auxiliary
losses or similar measures. The result is a highly interpretable generative
model on par with comparable auto-regressive models on the task of word
generation.
