  Panoptic narrative grounding (PNG) aims to segment things and stuff objects
in an image described by noun phrases of a narrative caption. As a multimodal
task, an essential aspect of PNG is the visual-linguistic interaction between
image and caption. The previous two-stage method aggregates visual contexts
from offline-generated mask proposals to phrase features, which tend to be
noisy and fragmentary. The recent one-stage method aggregates only pixel
contexts from image features to phrase features, which may incur semantic
misalignment due to lacking object priors. To realize more comprehensive
visual-linguistic interaction, we propose to enrich phrases with coupled pixel
and object contexts by designing a Phrase-Pixel-Object Transformer Decoder
(PPO-TD), where both fine-grained part details and coarse-grained entity clues
are aggregated to phrase features. In addition, we also propose a PhraseObject
Contrastive Loss (POCL) to pull closer the matched phrase-object pairs and push
away unmatched ones for aggregating more precise object contexts from more
phrase-relevant object tokens. Extensive experiments on the PNG benchmark show
our method achieves new state-of-the-art performance with large margins.
