  Accurate human trajectory prediction is crucial for applications such as
autonomous vehicles, robotics, and surveillance systems. Yet, existing models
often fail to fully leverage the non-verbal social cues human subconsciously
communicate when navigating the space. To address this, we introduce
Social-Transmotion, a generic Transformer-based model that exploits diverse and
numerous visual cues to predict human behavior. We translate the idea of a
prompt from Natural Language Processing (NLP) to the task of human trajectory
prediction, where a prompt can be a sequence of x-y coordinates on the ground,
bounding boxes in the image plane, or body pose keypoints in either 2D or 3D.
This, in turn, augments trajectory data, leading to enhanced human trajectory
prediction. Using masking technique, our model exhibits flexibility and
adaptability by capturing spatiotemporal interactions between agents based on
the available visual cues. We delve into the merits of using 2D versus 3D
poses, and a limited set of poses. Additionally, we investigate the spatial and
temporal attention map to identify which keypoints and time-steps in the
sequence are vital for optimizing human trajectory prediction. Our approach is
validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists
in Road Traffic, and ETH-UCY. The code is publicly available:
https://github.com/vita-epfl/social-transmotion.
