  Pedestrian trajectory prediction is an essential and challenging task for a
variety of real-life applications such as autonomous driving and robotic motion
planning. Besides generating a single future path, predicting multiple
plausible future paths is becoming popular in some recent work on trajectory
prediction. However, existing methods typically emphasize spatial interactions
between pedestrians and surrounding areas but ignore the smoothness and
temporal consistency of predictions. Our model aims to forecast multiple paths
based on a historical trajectory by modeling multi-scale graph-based spatial
transformers combined with a trajectory smoothing algorithm named ``Memory
Replay'' utilizing a memory graph. Our method can comprehensively exploit the
spatial information as well as correct the temporally inconsistent trajectories
(e.g., sharp turns). We also propose a new evaluation metric named ``Percentage
of Trajectory Usage'' to evaluate the comprehensiveness of diverse multi-future
predictions. Our extensive experiments show that the proposed model achieves
state-of-the-art performance on multi-future prediction and competitive results
for single-future prediction. Code released at
https://github.com/Jacobieee/ST-MR.
