  For autonomous agents to successfully operate in the real world, anticipation
of future events and states of their environment is a key competence. This
problem has been formalized as a sequence extrapolation problem, where a number
of observations are used to predict the sequence into the future. Real-world
scenarios demand a model of uncertainty of such predictions, as predictions
become increasingly uncertain -- in particular on long time horizons. While
impressive results have been shown on point estimates, scenarios that induce
multi-modal distributions over future sequences remain challenging. Our work
addresses these challenges in a Gaussian Latent Variable model for sequence
prediction. Our core contribution is a "Best of Many" sample objective that
leads to more accurate and more diverse predictions that better capture the
true variations in real-world sequence data. Beyond our analysis of improved
model fit, our models also empirically outperform prior work on three diverse
tasks ranging from traffic scenes to weather data.
