  Accurate long-term trajectory prediction in complex scenes, where multiple
agents (e.g., pedestrians or vehicles) interact with each other and the
environment while attempting to accomplish diverse and often unknown goals, is
a challenging stochastic forecasting problem. In this work, we propose MUSE, a
new probabilistic modeling framework based on a cascade of Conditional VAEs,
which tackles the long-term, uncertain trajectory prediction task using a
coarse-to-fine multi-factor forecasting architecture. In its Macro stage, the
model learns a joint pixel-space representation of two key factors, the
underlying environment and the agent movements, to predict the long and
short-term motion goals. Conditioned on them, the Micro stage learns a
fine-grained spatio-temporal representation for the prediction of individual
agent trajectories. The VAE backbones across the two stages make it possible to
naturally account for the joint uncertainty at both levels of granularity. As a
result, MUSE offers diverse and simultaneously more accurate predictions
compared to the current state-of-the-art. We demonstrate these assertions
through a comprehensive set of experiments on nuScenes and SDD benchmarks as
well as PFSD, a new synthetic dataset, which challenges the forecasting ability
of models on complex agent-environment interaction scenarios.
