  Multi-agent motion prediction is challenging because it aims to foresee the
future trajectories of multiple agents (\textit{e.g.} pedestrians)
simultaneously in a complicated scene. Existing work addressed this challenge
by either learning social spatial interactions represented by the positions of
a group of pedestrians, while ignoring their temporal coherence (\textit{i.e.}
dependencies between different long trajectories), or by understanding the
complicated scene layout (\textit{e.g.} scene segmentation) to ensure safe
navigation. However, unlike previous work that isolated the spatial
interaction, temporal coherence, and scene layout, this paper designs a new
mechanism, \textit{i.e.}, Dynamic and Static Context-aware Motion Predictor
(DSCMP), to integrates these rich information into the long-short-term-memory
(LSTM). It has three appealing benefits. (1) DSCMP models the dynamic
interactions between agents by learning both their spatial positions and
temporal coherence, as well as understanding the contextual scene layout.(2)
Different from previous LSTM models that predict motions by propagating hidden
features frame by frame, limiting the capacity to learn correlations between
long trajectories, we carefully design a differentiable queue mechanism in
DSCMP, which is able to explicitly memorize and learn the correlations between
long trajectories. (3) DSCMP captures the context of scene by inferring latent
variable, which enables multimodal predictions with meaningful semantic scene
layout. Extensive experiments show that DSCMP outperforms state-of-the-art
methods by large margins, such as 9.05\% and 7.62\% relative improvements on
the ETH-UCY and SDD datasets respectively.
