  Accurate prediction of others' trajectories is essential for autonomous
driving. Trajectory prediction is challenging because it requires reasoning
about agents' past movements, social interactions among varying numbers and
kinds of agents, constraints from the scene context, and the stochasticity of
human behavior. Our approach models these interactions and constraints jointly
within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the
model encodes multiple agents' past trajectories and the scene context into a
Multi-Agent Tensor, then applies convolutional fusion to capture multiagent
interactions while retaining the spatial structure of agents and the scene
context. The model decodes recurrently to multiple agents' future trajectories,
using adversarial loss to learn stochastic predictions. Experiments on both
highway driving and pedestrian crowd datasets show that the model achieves
state-of-the-art prediction accuracy.
