  We address the problem of forecasting pedestrian and vehicle trajectories in
unknown environments, conditioned on their past motion and scene structure.
Trajectory forecasting is a challenging problem due to the large variation in
scene structure and the multimodal distribution of future trajectories. Unlike
prior approaches that directly learn one-to-many mappings from observed context
to multiple future trajectories, we propose to condition trajectory forecasts
on plans sampled from a grid based policy learned using maximum entropy inverse
reinforcement learning (MaxEnt IRL). We reformulate MaxEnt IRL to allow the
policy to jointly infer plausible agent goals, and paths to those goals on a
coarse 2-D grid defined over the scene. We propose an attention based
trajectory generator that generates continuous valued future trajectories
conditioned on state sequences sampled from the MaxEnt policy. Quantitative and
qualitative evaluation on the publicly available Stanford drone and NuScenes
datasets shows that our model generates trajectories that are diverse,
representing the multimodal predictive distribution, and precise, conforming to
the underlying scene structure over long prediction horizons.
