  Referring expression grounding is an important and challenging task in
computer vision. To avoid the laborious annotation in conventional referring
grounding, unpaired referring grounding is introduced, where the training data
only contains a number of images and queries without correspondences. The few
existing solutions to unpaired referring grounding are still preliminary, due
to the challenges of learning image-text matching and lack of the top-down
guidance with unpaired data. In this paper, we propose a novel bidirectional
cross-modal matching (BiCM) framework to address these challenges.
Particularly, we design a query-aware attention map (QAM) module that
introduces top-down perspective via generating query-specific visual attention
maps. A cross-modal object matching (COM) module is further introduced, which
exploits the recently emerged image-text matching pretrained model, CLIP, to
predict the target objects from a bottom-up perspective. The top-down and
bottom-up predictions are then integrated via a similarity funsion (SF) module.
We also propose a knowledge adaptation matching (KAM) module that leverages
unpaired training data to adapt pretrained knowledge to the target dataset and
task. Experiments show that our framework outperforms previous works by 6.55%
and 9.94% on two popular grounding datasets.
