  Ambiguity is inherently present in many machine learning tasks, but
especially for sequential models seldom accounted for, as most only output a
single prediction. In this work we propose an extension of the Multiple
Hypothesis Prediction (MHP) model to handle ambiguous predictions with
sequential data, which is of special importance, as often multiple futures are
equally likely. Our approach can be applied to the most common recurrent
architectures and can be used with any loss function. Additionally, we
introduce a novel metric for ambiguous problems, which is better suited to
account for uncertainties and coincides with our intuitive understanding of
correctness in the presence of multiple labels. We test our method on several
experiments and across diverse tasks dealing with time series data, such as
trajectory forecasting and maneuver prediction, achieving promising results.
