  Video Object Grounding (VOG) is the problem of associating spatial object
regions in the video to a descriptive natural language query. This is a
challenging vision-language task that necessitates constructing the correct
cross-modal correspondence and modeling the appropriate spatio-temporal context
of the query video and caption, thereby localizing the specific objects
accurately. In this paper, we tackle this task by a novel framework called
HiErarchical spatio-tempoRal reasOning (HERO) with contrastive action
correspondence. We study the VOG task at two aspects that prior works
overlooked: (1) Contrastive Action Correspondence-aware Retrieval. Notice that
the fine-grained video semantics (e.g., multiple actions) is not totally
aligned with the annotated language query (e.g., single action), we first
introduce the weakly-supervised contrastive learning that classifies the video
as action-consistent and action-independent frames relying on the video-caption
action semantic correspondence. Such a design can build the fine-grained
cross-modal correspondence for more accurate subsequent VOG. (2) Hierarchical
Spatio-temporal Modeling Improvement. While transformer-based VOG models
present their potential in sequential modality (i.e., video and caption)
modeling, existing evidence also indicates that the transformer suffers from
the issue of the insensitive spatio-temporal locality. Motivated by that, we
carefully design the hierarchical reasoning layers to decouple fully connected
multi-head attention and remove the redundant interfering correlations.
Furthermore, our proposed pyramid and shifted alignment mechanisms are
effective to improve the cross-modal information utilization of neighborhood
spatial regions and temporal frames. We conducted extensive experiments to show
our HERO outperforms existing techniques by achieving significant improvement
on two benchmark datasets.
