  Forecasting the future states of surrounding traffic participants is a
crucial capability for autonomous vehicles. The recently proposed occupancy
flow field prediction introduces a scalable and effective representation to
jointly predict surrounding agents' future motions in a scene. However, the
challenging part is to model the underlying social interactions among traffic
agents and the relations between occupancy and flow. Therefore, this paper
proposes a novel Multi-modal Hierarchical Transformer network that fuses the
vectorized (agent motion) and visual (scene flow, map, and occupancy)
modalities and jointly predicts the flow and occupancy of the scene.
Specifically, visual and vector features from sensory data are encoded through
a multi-stage Transformer module and then a late-fusion Transformer module with
temporal pixel-wise attention. Importantly, a flow-guided multi-head
self-attention (FG-MSA) module is designed to better aggregate the information
on occupancy and flow and model the mathematical relations between them. The
proposed method is comprehensively validated on the Waymo Open Motion Dataset
and compared against several state-of-the-art models. The results reveal that
our model with much more compact architecture and data inputs than other
methods can achieve comparable performance. We also demonstrate the
effectiveness of incorporating vectorized agent motion features and the
proposed FG-MSA module. Compared to the ablated model without the FG-MSA
module, which won 2nd place in the 2022 Waymo Occupancy and Flow Prediction
Challenge, the current model shows better separability for flow and occupancy
and further performance improvements.
