  Multi-modal trajectory forecasting methods commonly evaluate using
single-agent metrics (marginal metrics), such as minimum Average Displacement
Error (ADE) and Final Displacement Error (FDE), which fail to capture joint
performance of multiple interacting agents. Only focusing on marginal metrics
can lead to unnatural predictions, such as colliding trajectories or diverging
trajectories for people who are clearly walking together as a group.
Consequently, methods optimized for marginal metrics lead to overly-optimistic
estimations of performance, which is detrimental to progress in trajectory
forecasting research. In response to the limitations of marginal metrics, we
present the first comprehensive evaluation of state-of-the-art (SOTA)
trajectory forecasting methods with respect to multi-agent metrics (joint
metrics): JADE, JFDE, and collision rate. We demonstrate the importance of
joint metrics as opposed to marginal metrics with quantitative evidence and
qualitative examples drawn from the ETH / UCY and Stanford Drone datasets. We
introduce a new loss function incorporating joint metrics that, when applied to
a SOTA trajectory forecasting method, achieves a 7\% improvement in JADE / JFDE
on the ETH / UCY datasets with respect to the previous SOTA. Our results also
indicate that optimizing for joint metrics naturally leads to an improvement in
interaction modeling, as evidenced by a 16\% decrease in mean collision rate on
the ETH / UCY datasets with respect to the previous SOTA. Code is available at
\texttt{\hyperlink{https://github.com/ericaweng/joint-metrics-matter}{github.com/ericaweng/joint-metrics-matter}}.
