  For prediction of interacting agents' trajectories, we propose an end-to-end
trainable architecture that hybridizes neural nets with game-theoretic
reasoning, has interpretable intermediate representations, and transfers to
downstream decision making. It uses a net that reveals preferences from the
agents' past joint trajectory, and a differentiable implicit layer that maps
these preferences to local Nash equilibria, forming the modes of the predicted
future trajectory. Additionally, it learns an equilibrium refinement concept.
For tractability, we introduce a new class of continuous potential games and an
equilibrium-separating partition of the action space. We provide theoretical
results for explicit gradients and soundness. In experiments, we evaluate our
approach on two real-world data sets, where we predict highway driver merging
trajectories, and on a simple decision-making transfer task.
