  Understanding multi-agent behavior is critical across various fields. The
conventional approach involves analyzing agent movements through three primary
tasks: trajectory prediction, imputation, and spatial-temporal recovery.
Considering the unique input formulation and constraint of these tasks, most
existing methods are tailored to address only one specific task. However, in
real-world applications, these scenarios frequently occur simultaneously.
Consequently, methods designed for one task often fail to adapt to others,
resulting in performance drops. To overcome this limitation, we propose a
Unified Trajectory Generation model, UniTraj, that processes arbitrary
trajectories as masked inputs, adaptable to diverse scenarios. Specifically, we
introduce a Ghost Spatial Masking (GSM) module embedded within a Transformer
encoder for spatial feature extraction. We further extend recent successful
State Space Models (SSMs), particularly the Mamba model, into a Bidirectional
Temporal Mamba to effectively capture temporal dependencies. Additionally, we
incorporate a Bidirectional Temporal Scaled (BTS) module to comprehensively
scan trajectories while maintaining the temporal missing relationships within
the sequence. We curate and benchmark three practical sports game datasets,
Basketball-U, Football-U, and Soccer-U, for evaluation. Extensive experiments
demonstrate the superior performance of our model. To the best of our
knowledge, this is the first work that addresses this unified problem through a
versatile generative framework, thereby enhancing our understanding of
multi-agent movement. Our datasets, code, and model weights are available at
https://github.com/colorfulfuture/UniTraj-pytorch.
