  Understanding and forecasting future trajectories of agents are critical for
behavior analysis, robot navigation, autonomous cars, and other related
applications. Previous methods mostly treat trajectory prediction as time
sequence generation. Different from them, this work studies agents'
trajectories in a "vertical" view, i.e., modeling and forecasting trajectories
from the spectral domain. Different frequency bands in the trajectory spectrums
could hierarchically reflect agents' motion preferences at different scales.
The low-frequency and high-frequency portions could represent their coarse
motion trends and fine motion variations, respectively. Accordingly, we propose
a hierarchical network V$^2$-Net, which contains two sub-networks, to
hierarchically model and predict agents' trajectories with trajectory
spectrums. The coarse-level keypoints estimation sub-network first predicts the
"minimal" spectrums of agents' trajectories on several "key" frequency
portions. Then the fine-level spectrum interpolation sub-network interpolates
the spectrums to reconstruct the final predictions. Experimental results
display the competitiveness and superiority of V$^2$-Net on both ETH-UCY
benchmark and the Stanford Drone Dataset.
