  Multi-agent interactions are important to model for forecasting other agents'
behaviors and trajectories. At a certain time, to forecast a reasonable future
trajectory, each agent needs to pay attention to the interactions with only a
small group of most relevant agents instead of unnecessarily paying attention
to all the other agents. However, existing attention modeling works ignore that
human attention in driving does not change rapidly, and may introduce
fluctuating attention across time steps. In this paper, we formulate an
attention model for multi-agent interactions based on a total variation
temporal smoothness prior and propose a trajectory prediction architecture that
leverages the knowledge of these attended interactions. We demonstrate how the
total variation attention prior along with the new sequence prediction loss
terms leads to smoother attention and more sample-efficient learning of
multi-agent trajectory prediction, and show its advantages in terms of
prediction accuracy by comparing it with the state-of-the-art approaches on
both synthetic and naturalistic driving data. We demonstrate the performance of
our algorithm for trajectory prediction on the INTERACTION dataset on our
website.
