  Systems consisting of interacting agents are prevalent in the world, ranging
from dynamical systems in physics to complex biological networks. To build
systems which can interact robustly in the real world, it is thus important to
be able to infer the precise interactions governing such systems. Existing
approaches typically discover such interactions by explicitly modeling the
feed-forward dynamics of the trajectories. In this work, we propose Neural
Interaction Inference with Potentials (NIIP) as an alternative approach to
discover such interactions that enables greater flexibility in trajectory
modeling: it discovers a set of relational potentials, represented as energy
functions, which when minimized reconstruct the original trajectory. NIIP
assigns low energy to the subset of trajectories which respect the relational
constraints observed. We illustrate that with these representations NIIP
displays unique capabilities in test-time. First, it allows trajectory
manipulation, such as interchanging interaction types across separately trained
models, as well as trajectory forecasting. Additionally, it allows adding
external hand-crafted potentials at test-time. Finally, NIIP enables the
detection of out-of-distribution samples and anomalies without explicit
training. Website: https://energy-based-model.github.io/interaction-potentials.
