  Predicting future trajectories is critical in autonomous navigation,
especially in preventing accidents involving humans, where a predictive agent's
ability to anticipate in advance is of utmost importance. Trajectory
forecasting models, employed in fields such as robotics, autonomous vehicles,
and navigation, face challenges in real-world scenarios, often due to the
isolation of model components. To address this, we introduce a novel dataset
for end-to-end trajectory forecasting, facilitating the evaluation of models in
scenarios involving less-than-ideal preceding modules such as tracking. This
dataset, an extension of the JRDB dataset, provides comprehensive data,
including the locations of all agents, scene images, and point clouds, all from
the robot's perspective. The objective is to predict the future positions of
agents relative to the robot using raw sensory input data. It bridges the gap
between isolated models and practical applications, promoting a deeper
understanding of navigation dynamics. Additionally, we introduce a novel metric
for assessing trajectory forecasting models in real-world scenarios where
ground-truth identities are inaccessible, addressing issues related to
undetected or over-detected agents. Researchers are encouraged to use our
benchmark for model evaluation and benchmarking.
