  Trajectory prediction is a critical functionality of autonomous systems that
share environments with uncontrolled agents, one prominent example being
self-driving vehicles. Currently, most prediction methods do not enforce scene
consistency, i.e., there are a substantial amount of self-collisions between
predicted trajectories of different agents in the scene. Moreover, many
approaches generate individual trajectory predictions per agent instead of
joint trajectory predictions of the whole scene, which makes downstream
planning difficult. In this work, we present ScePT, a policy planning-based
trajectory prediction model that generates accurate, scene-consistent
trajectory predictions suitable for autonomous system motion planning. It
explicitly enforces scene consistency and learns an agent interaction policy
that can be used for conditional prediction. Experiments on multiple real-world
pedestrians and autonomous vehicle datasets show that ScePT} matches current
state-of-the-art prediction accuracy with significantly improved scene
consistency. We also demonstrate ScePT's ability to work with a downstream
contingency planner.
