  Panoptic Narrative Grounding (PNG) is an emerging cross-modal grounding task,
which locates the target regions of an image corresponding to the text
description. Existing approaches for PNG are mainly based on a two-stage
paradigm, which is computationally expensive. In this paper, we propose a
one-stage network for real-time PNG, termed End-to-End Panoptic Narrative
Grounding network (EPNG), which directly generates masks for referents.
Specifically, we propose two innovative designs, i.e., Locality-Perceptive
Attention (LPA) and a bidirectional Semantic Alignment Loss (SAL), to properly
handle the many-to-many relationship between textual expressions and visual
objects. LPA embeds the local spatial priors into attention modeling, i.e., a
pixel may belong to multiple masks at different scales, thereby improving
segmentation. To help understand the complex semantic relationships, SAL
proposes a bidirectional contrastive objective to regularize the semantic
consistency inter modalities. Extensive experiments on the PNG benchmark
dataset demonstrate the effectiveness and efficiency of our method. Compared to
the single-stage baseline, our method achieves a significant improvement of up
to 9.4% accuracy. More importantly, our EPNG is 10 times faster than the
two-stage model. Meanwhile, the generalization ability of EPNG is also
validated by zero-shot experiments on other grounding tasks.
