  Recent advances in trajectory prediction have shown that explicit reasoning
about agents' intent is important to accurately forecast their motion. However,
the current research activities are not directly applicable to intelligent and
safety critical systems. This is mainly because very few public datasets are
available, and they only consider pedestrian-specific intents for a short
temporal horizon from a restricted egocentric view. To this end, we propose
LOKI (LOng term and Key Intentions), a novel large-scale dataset that is
designed to tackle joint trajectory and intention prediction for heterogeneous
traffic agents (pedestrians and vehicles) in an autonomous driving setting. The
LOKI dataset is created to discover several factors that may affect intention,
including i) agent's own will, ii) social interactions, iii) environmental
constraints, and iv) contextual information. We also propose a model that
jointly performs trajectory and intention prediction, showing that recurrently
reasoning about intention can assist with trajectory prediction. We show our
method outperforms state-of-the-art trajectory prediction methods by upto
$27\%$ and also provide a baseline for frame-wise intention estimation.
