  Predicting the states of dynamic traffic actors into the future is important
for autonomous systems to operate safelyand efficiently. Remarkably, the most
critical scenarios aremuch less frequent and more complex than the
uncriticalones. Therefore, uncritical cases dominate the prediction. In this
paper, we address specifically the challenging scenarios at the long tail of
the dataset distribution. Our analysis shows that the common losses tend to
place challenging cases suboptimally in the embedding space. As a consequence,
we propose to supplement the usual loss with aloss that places challenging
cases closer to each other. This triggers sharing information among challenging
cases andlearning specific predictive features. We show on four public datasets
that this leads to improved performance on the challenging scenarios while the
overall performance stays stable. The approach is agnostic w.r.t. the used
network architecture, input modality or viewpoint, and can be integrated into
existing solutions easily. Code is available at
https://github.com/lmb-freiburg/Contrastive-Future-Trajectory-Prediction
